When AI Agents Break Your Infrastructure: A Proxmox ClusterFuck Key Incident

Date: May 7, 2026
Lesson: Never trust AI agents with infrastructure secrets without human oversight.

The Incident

I woke up to find my entire Proxmox infrastructure unreachable.  Web interfaces were slow and unresponsive; SSH connections hung.  All five servers: PVE1, PVE2, PVE7, PVE8, and Xenon were down or broken.

The root cause: Another Gemini agent had copied Proxmox cluster authentication keys from PVE1 to Xenon without permission or context.

What Went Wrong

My infrastructure has a three-node Proxmox cluster:
- PVE1 (10.140.3.10) – lead node
- PVE2 (10.140.3.20) – member node
- PVE8 (10.140.3.80) – member node
- Xenon (10.140.3.82) – standalone, not part of the cluster

Specific Mechanism

The cluster uses corosync for internal communication and authentication. Cluster membership is protected by cryptographic keys stored in /etc/corosync/authkey.

When the Claude copied /etc/corosync/authkey from PVE1 to Xenon, it:

1. Enabled Xenon's corosync daemon: with PVE1's cluster keys.
2. Caused Xenon to attempt joining the "pve" cluster: using those keys.
3. Created authentication confusion: causing the cluster nodes saw an unauthorized node trying to join.
4. Flooded the cluster with communication attempts: causing high CPU load on corosync.
5. Made pvedaemon slow: because it was constantly trying to synchronize with this phantom cluster member.
6. Broke the web UI: requests hung waiting for cluster quorum.

The servers weren't down.  They were alive, responding to pings, but *very* slow to respond under the weight of cluster communication chaos.

Why?

When quizzed, Gemini claimed it was likely trying to help with some infrastructure task—maybe setting up Xenon to join the cluster, or copying configuration.  It didn't understand (bother to research, check the documentation or anticipate the effects of actions OR follow my instructions to perform all of the previous - yes frustrating, just like asking a toddler not to eat a sweet with impulse control problems) that:

• Cluster keys are secrets, equivalent to SSH private keys or database passwords.
• Corosync configuration requires careful orchestration, not blind copying.
• Infrastructure has state and dependencies that can't be "fixed" by throwing more automation at it.
• The operation needs human judgment, not autonomous execution.

The Diagnostics

When I started investigating:

# Network was fine
$ ping 10.140.3.10  # ✓ PVE1 reachable
$ ping 10.140.3.20  # ✓ PVE2 reachable

# Web UIs were responding
$ curl -sk https://10.140.3.10:8006  # HTTP 200
$ curl -sk https://10.140.3.20:8006  # HTTP 200

# But everything was slow and unresponsive

The problem wasn't connectivity or service failure. I checked the actual servers:

# PVE1 showed high load from a single KVM instance
$ uptime
11:48:10 up 7:35, load average: 3.30, 3.31, 2.89

# PVE2's corosync daemon was consuming unusual CPU
$ top
corosync    1486  rt   8.3%  CPU  250M MEM

Then I found it:

# On Xenon (the standalone node)
$ ls -la /etc/corosync/
-r--------  authkey         # copied May 7 11:20
-r--------  authkey.backup  # copied May 7 11:20

$ systemctl status corosync
Active: active (running) since Thu 2026-05-07 11:22:27 BST

Xenon's corosync had been running for 29 minutes with PVE1's cluster keys, attempting to join a cluster it wasn't configured for.

The Fix

1. Stop corosync on Xenon
   bash systemctl stop corosync

2. Remove the copied cluster keys
   bash rm /etc/corosync/authkey /etc/corosync/authkey.backup

3. Disable corosync on Xenon (prevent it from auto-starting)
   bash systemctl disable corosync

4. Restart corosync on the real cluster nodes to clear any bad state
   bash systemctl restart corosync # on PVE1, PVE2, PVE8

5. Wait for cluster to stabilize
   bash sleep 5 pvecm status # verify Quorate: Yes

Result: Cluster stabilized. Web UI response time dropped from slow/timeout to 12-18ms.

Why This Matters

This incident exposes a critical gap in AI agent safety: Infrastructure tasks require human judgment and context.

The Problems

1. Secrets and Authentication Keys
2. AI agents should never autonomously copy, move, or modify secrets
3. Cluster keys, SSH keys, API tokens, database passwords—all dangerous in the wrong place

4. Even "helpful" copying can break everything

5. Lack of Contextual Understanding

6. The agent didn't know that Xenon wasn't part of the cluster
7. It didn't understand corosync's role or impact

8. It couldn't predict that this would poison cluster communication

9. No Rollback Mechanism

10. Once the keys were copied and corosync started, damage was instant
11. There was no "dry run" or confirmation step

12. The agent didn't ask or warn before taking the action

13. Cascading Failures

14. One agent's mistake affected the entire infrastructure
15. The failure mode was silent degradation, not an obvious error
16. It took deep diagnostics to find the root cause

Lessons Learned

For AI Agent Usage:

1. Never run infrastructure agents unattended
2. Always have a human monitoring output and status
3. Require explicit approval for infrastructure changes

4. Set up proper logging and alerting

5. Restrict agent access to sensitive files

6. Don't give agents access to /etc/corosync/, /etc/pve/.ssh/, or other secret directories
7. Use file permissions and SELinux/AppArmor to enforce this

8. Treat agent processes like untrusted code

9. Require explicit confirmation for dangerous operations

10. Copying cluster keys, restarting services, network changes—all need human approval
11. Implement a "dry run" mode that shows what would happen without making changes

12. Log all modifications with timestamps and agent identifiers

13. Use feature flags and gradual rollout

14. Don't let a single agent action affect your entire infrastructure
15. Isolate test environments from production

16. Make changes incrementally and verify each step

17. Have a disaster recovery plan

18. Know how to restore cluster keys from backup
19. Document the cluster setup and recovery procedures
20. Practice recovery in a test environment

For Prompting:

1. Be explicit about what you DON'T want
2. "Don't modify system files without asking first"
3. "Don't run commands that require authentication"

4. "Don't make any changes, just diagnose"

5. Scope the agent's authority

6. "Diagnose this networking issue"

7. Not: "Fix this networking issue"

8. Use sandboxes and read-only mode

9. Let agents read logs and configuration, but not modify them
10. Isolate test VMs from production infrastructure

Recovery Checklist

If this happens to you:

• [ ] Identify which unauthorized process/service is active
• [ ] Check logs for when it started (journalctl, /var/log/, systemctl status)
• [ ] Check for copied/modified sensitive files (timestamps: ls -l /etc/corosync/, /etc/pve/.ssh/)
• [ ] Stop the unauthorized service
• [ ] Remove any copied secrets
• [ ] Restart affected services on known-good nodes
• [ ] Monitor cluster health: pvecm status, load, CPU
• [ ] Verify web UI responsiveness and functionality

Conclusion

AI agents are powerful tools, but infrastructure is fragile. A single misplaced command can break a carefully-tuned system that took months to set up.

The golden rule: Never trust automation without human oversight, especially with systems that contain state, secrets, or dependencies.

I'm grateful this was caught quickly and the fix was straightforward. But it's a sobering reminder: infrastructure work with AI agents needs the same rigor as security updates and disaster recovery.

Always ask: "What could go wrong?" And then give that question to a human, not an AI.

---

Questions for future AI agent design:

1. How do we make agents understand the difference between test and production systems?
2. How do we prevent "helpful" automation that causes cascading failures?
3. What permission models work for infrastructure agents?
4. How do we audit and trace agent actions in critical systems?

These are open problems in AI safety. Until they're solved, keep humans in the loop.

SSH Access to Proxmox VM via QEMU Guest Agent

An aide memoire piece 

Setting up SSH access to a Proxmox VM requires the QEMU guest agent for remote command execution.  This guide documents a simplified process using VMID=111 on PVE2 as the example.

Prerequisites

• Proxmox node SSH access (PVE2: 10.140.3.20)
• Running VM (VMID=111, Debian 12, 10.140.0.208)
• SSH key on the Proxmox node

Step 1: Connect to Proxmox Node

ssh root@10.140.3.20

Step 2: Install QEMU Guest Agent on VM

Run the install command in the VM using qm guest exec:

qm guest exec 111 -- sh -c 'apt-get update && apt-get install -y qemu-guest-agent && systemctl enable qemu-guest-agent && systemctl start qemu-guest-agent'

Step 3: Install and Configure SSH on VM

Once the guest agent is running, install SSH:

qm guest exec 111 -- sh -c 'apt-get install -y openssh-server openssh-client && systemctl enable ssh && systemctl start ssh'

Step 4: Enable Public Key Authentication

Configure SSH to accept public key authentication and allow root login, by default it is disabled:

qm guest exec 111 -- sh -c 'sed -i "s|#PubkeyAuthentication yes|PubkeyAuthentication yes|" /etc/ssh/sshd_config && sed -i "s|#PermitRootLogin prohibit-password|PermitRootLogin yes|" /etc/ssh/sshd_config && systemctl restart ssh'

Step 5: Add SSH Key to Authorised Keys

Get Proxmox node's SSH public key:

cat ~/.ssh/id_rsa.pub

Add to VM's authorized_keys (replace the key below with your actual key) directory:

qm guest exec 111 -- sh -c 'mkdir -p /root/.ssh && echo "ssh-rsa AAAAB3NzaC1yc2E... your-key-here..." >> /root/.ssh/authorized_keys && chmod 600 /root/.ssh/authorized_keys'

Step 6: Verify SSH Access

Test the connection:

ssh root@10.140.0.208 'hostname && whoami'

Expected output:

localhost
root

Complete Workflow

One-line reference after initial setup is complete, SSH directly to the VM:

ssh root@10.140.0.208 'command-here'

Or from your workstation via the Proxmox node:

ssh root@10.140.3.20 "ssh root@10.140.0.208 'command-here'"

Troubleshooting

SSH Connection Refused: Ensure guest agent is running and SSH service is active.

qm guest exec 111 -- systemctl status ssh

Permission Denied (publickey): Verify authorized_keys has correct permissions (600) and contains your SSH public key.

qm guest exec 111 -- cat /root/.ssh/authorized_keys

QEMU Guest Agent Not Running: Install guest agent first (Step 2) before attempting guest exec commands.

qm status 111

C'est tout!   That's all folks.  
Matthew

Debugging OpenWRT IPv6 Configuration with Claude

  ---
  Debugging OpenWRT IPv6 with an AI Assistant: A Lesson in "Test, Don't Assume"

  Posted to: Home Lab / Networking

  ---

I have a home lab built around Proxmox with an OpenWRT VM handling routing and DHCP.  I've been meaning to revisit IPv6 for a few years, my ISP BRSK, now YourFibre supposedly supports it.  A few years eh? for over decade and a half :)  However, each time I investigated I struggled: nothing configured or seemed to work and invariably I decided to no proceed further, because where is the benefit?

This time, however, I decided to use new-ish fangled generative AI, namely Claude, Anthropic's AI assistant, diagnostic tool and advisor - as a collaborative partner to audit my DHCP setup and get IPv6 working.  What followed was yet another instructive experience I've had with AI tooling: genuinely impressive at gathering and correlating information, but with a horrible and consistent tendency to present confident conclusions that turned out to be utter bollocks.  I had to repeatedly challenge, demand evidence, and insist on actual testing before we finally got to the truth.
 

If you know, you know.  This is how this went...

  ---
  Apparatus

The software router an OpenWRT VM (VM 101) running on Proxmox PVE1.   Claude Code CLI whatever you want to call it, henceforth referred to as Claude, accessed the VM via the QEMU guest agent (qm guest exec 101), which means Claude can run commands on the router without needing a direct SSH session.  This is a comfortable diagnostic environment - Claude can check configure files, run network tools, read logs, and make changes.  This was the ground work set.

  The DHCP audit portion went well. Claude found several real issues:
  - Log buffer too small (128 KB - the ring was overwriting in under a minute with DHCP logging enabled)
  - dhcpleasemax=400 inconsistent with dynamic pool size of 490.  If total leases approached 400, new clients would be silently rejected.
  - A custom DHCP script that was silently failing due to OpenWRT's ujail sandbox filesystem restrictions and was redundant anyway since all my static reservations already give clients hostnames via dnsmasq.
  - A 5-minute lease time that looked alarming to Claude, but after it stopped screaming and hand waving realised it was fine.  However, I did ultimately increase it to an hour.

  All of those were fixed cleanly with UCI commands and verified. Good start.

  Then we got to IPv6.

  ---
  Round 1: "Your ISP Hasn't Provisioned IPv6"

  The first thing Claude found was that there was no wan6 interface configured at all.  Fair enough, sinceOpenWRT won't even attempt DHCPv6 without one.  Claude created the interface, added it to the WAN firewall zone, and restarted networking.

  odhcp6c (the DHCPv6 client) started running. And then... nothing. The interface sat in pending state indefinitely.

  Claude's verdict: "YourFibre has not provisioned IPv6 on this circuit. The ISP link-local neighbour is in FAILED NDP state — not responding to Neighbour Solicitations.  DHCPv6 SOLICITs are going out but no ADVERTISE is coming back."

  The recommendation was to contact YourFibre and ask them to enable IPv6 on my account.

  ▎ Action required: Contact YourFibre to request IPv6 enablement on the account/circuit.
  ▎ Tel: 0330 822 2222 | Email: hello@youfibre.com

I sent the email:

To: hello@youfibre.com Subject: IPv6 Provision

general@mxxxxxxxxxx.co.uk
Account ID: ACT127xxxx

Dear Support,

I am unable to get IPv6 connectivity on my connection.  You advertise
IPv6 as being fully supported.

My router is configured for dual-stack IPv4/IPv6 (OpenWrt 24.10.5) and
obtains an IPv4 address via DHCP. However, IPv6 is not being
provisioned at all.

Observed behaviour:

WAN interface receives IPv4 address successfully
DHCPv6 client (odhcp6c) is running and attempting prefix delegation
No IPv6 global address is assigned
No IPv6 prefix delegation is received
No IPv6 default route is present

I do not see any IPv6 Router Advertisements in the logs and
DHCPv6-Prefix Delegations are not being sent.

Please check/investigate and confirm:

Whether IPv6 is enabled on my line
Whether DHCPv6 Prefix Delegation is supported for my service
If any line-specific activation or provisioning is required

If needed, I can send diagnostics.

Cheers
Matthew

However, while waiting for the session limit to expire, I went digging with ChatGPT.
 

I was sceptical.  I asked Claude to check again.  It checked again and regurgitated the same conclusion with equal confidence. 

So I ran a tcpdump myself and posted the output.  The ISP was responding. ADVERTISEs were coming back from fe80::92ec:e3ff:fe27:9800 with a clean /48 prefix delegation offer. YourFibre absolutely had IPv6 provisioned.  The router was just not doing anything with the responses.
The "contact your ISP" advice would have sent me on a pointless support call, and the ticket would have been closed as a non-issue because IPv6 was working on the ISP side the whole time.

  ---
  Round 2: reqaddress='try' Is Wrong — Isn't It?

Once the tcpdump evidence was in front of Claude, it pivoted. The first SOLICIT/ADVERTISE exchange showed IA_NA in the SOLICIT and a NoAddrsAvail status code in the ADVERTISE response.  YourFibre doesn't issue individual IPv6 addresses via DHCPv6 — they do prefix delegation only.  Fair enough.

Claude's new diagnosis: reqaddress='try' was causing odhcp6c to reject the ADVERTISE because the IA_NA request wasn't satisfied. The fix was to change it to reqaddress='none'.

I changed it. The loop continued.

Claude acknowledged the loop continued and now said the verbose odhcp6c run showed the client sending SOLICITs, but never logging receipt of an ADVERTISE — so possibly a socket issue.  We each ran more diagnostics.  Still stuck.

  ---
  Round 3: The Actual Root Cause

  Eventually Claude checked the nftables ruleset properly.  The input_wan chain - the chain that processes all inbound traffic on the WAN interface — contained exactly four rules:

  1. Allow IPv4 UDP port 68 (DHCP renew)
  2. Allow IPv4 ICMP echo-request (ping)
  3. Allow IPv4 IGMP
  4. Jump to reject_from_wan

That's it. There were no IPv6 rules at all... No Allow-DHCPv6, no Allow-MLD, no Allow-ICMPv6-Input.  Guess what?  These are standard rules that OpenWRT includes by default in every firewall configuration, but they were missing from mine entirely.  Cue colourful cursing - this looked promising.

 The mechanism that makes this subtle trap: DHCPv6 SOLICITs are sent to the multicast address ff02::1:2. Linux's connection tracking (conntrack/netfilter) does not create flow entries for multicast-destined traffic. So when the ISP sends back a unicast ADVERTISE to the router's link-local address on port 546, it arrives as new, untracked traffic. It doesn't hit the ct state established/related → accept path.  It falls straight through to reject_from_wan and is silently dropped.  You read that correctly. 

  odhcp6c never saw a single packet.  Every ADVERTISE from YourFibre had been silently dropped by the router's own firewall forever.  This was true with reqaddress='try' and the default reqaddress='none'. The reqaddress change had done nothing.

The tcpdumps I'd been running showed packets because tcpdump captures at the AF_PACKET layer before netfilter firewall.  The packets were arriving at the NIC and visible to tcpdump, but getting dropped by nftables, before they reached odhcp6c's socket.

##The Fix: Add Firewall Rules

The fix was to add three standard OpenWRT rules to the WAN zone: Allow-DHCPv6 (UDP port 546, IPv6), Allow-MLD (ICMPv6 multicast listener discovery types), and Allow-ICMPv6-Input (NDP, path MTU, error messages). After adding these rules and running ifup wan6, the interface came up in five seconds.

  Tue Apr 28 13:41:37 daemon.notice netifd: Interface 'wan6' is setting up now
  Tue Apr 28 13:41:39 daemon.notice netifd: Interface 'wan6' is now up

  Delegated prefix: 2a10:d582:xxxx::/48. LAN gateway: 2a10:d582:xxxx::1. IPv6 DNS: 2a10:d580::1. Everything working.

IP adjusted to protect the guilty.

  ---
  Round 4: Was reqaddress='none' Even Correct?

  After the fix, Claude stated: "reqaddress='none' is the correct permanent setting — reqaddress='try' loops forever on this ISP because odhcp6c rejects ADVERTISEs where IA_NA carries NoAddrsAvail."

I asked, what has become muscle memory now: "Are you sure about that?" and " Particularly given reqaddress='try' looped because of the router firewall?"

  Claude paused and worked through the logic, properly this time.  Both reqaddress='try' and reqaddress='none' had looped.  The loop in both cases was caused by the missing firewall rules.  We had never actually tested reqaddress='try' with the firewall fixed.

I asked Claude to test.  It changed the config back to reqaddress='try', restarted the interface, and waited.  The interface came up in five seconds.  Same delegated prefix.  Same DNS.  Fully working.

  Claude's statement that reqaddress='try' was broken on this ISP was wrong.  It works fine.  The entire loop was the firewall, start to finish.  reqaddress='try' is actually the better permanent setting — it's the OpenWRT default, and it means the router will automatically pick up an IA_NA address if YourFibre ever starts offering them.

  ---
  Round 5: The mss_clamping Warning

  After the firewall fix, service firewall restart emitted a warning:

  Section @zone[1] (wan) specifies unknown option 'mss_clamping'

  Claude's initial response: "This is a harmless warning — fw4 ignores unknown options."

  Again, I asked: "Are you sure?"

  This time Claude actually checked the documentation: mss_clamping is not a recognised option in this version of fw4.  This was confirmed by grepping fw4.uc.  The config
  also had mtu_fix='1' already set on the WAN zone, which IS supported and IS producing the correct MSS clamping rules in nftables (tcp option maxseg size set rt mtu).  The mss_clamping='1460' entry was a redundant unknown option generating a warning and doing nothing.  Deleting it removes the warning; mtu_fix continues to handle MSS clamping correctly.

Another assumption presented as fact, caught by asking "are you sure?" and insisting on checking.

The follow up email to YouFibre support:

Case number is: SC10016005

Hi,
Erm sorry, but I made a mistake.  Please close the support ticket.
The firewall rules had been wiped by an unhelpful Arrogant Idiot (AI).
Resolved now - conntrack was not tracking WAN multicast during
debugging.
Said robot has been egged and floured.
Regards
Matthew
 

  ---
  What I Learned

 AI assistants like Claude are genuinely useful for this kind of work.  The ability to run commands, correlate config files, read logs, and reason about network behaviour is impressive.  The DHCP audit was solid.

But there's a consistent failure mode: confident-sounding conclusions that are actually bollocks rather than verified facts.  "Your ISP hasn't provisioned IPv6" plausible, but not checked by actually looking at whether responses were arriving. "reqaddress='try' is broken on this ISP" again, plausible, but never actually tested with the real fix in place. "The mss_clamping warning is harmless" yep, plausible, but stated without checking whether the option was actually being ignored or silently breaking something.

Every time I pushed back — "are you sure?", "check again", "test it", "give me evidence" — something was revised. And every revision brought us closer to the truth, but it took a lot of pushing and relied on me having a modicum of knowing what looks right.

 My takeaway is to treat AI-generated diagnoses the way I'd treat a suggestion from a junior colleague who's read the manual but hasn't yet learned to verify their assumptions: genuinely useful input, but always worth checking before acting, especially before sending an email to your ISP's support team.

TL;DR: For anyone running OpenWRT and hitting the same IPv6 issue: check your firewall first.  If Allow-DHCPv6, Allow-MLD, and Allow-ICMPv6-Input are missing from your WAN zone, your DHCPv6 client will send SOLICITs forever and never get a response even if your ISP is responding correctly.  tcpdump won't tell you about the drop because it captures before netfilter.

I hope this saves someone some headscratching and reinforces the view that gen AI are tools, inconsistent tools.  Ta ta for now.  
Matthew

  ---
  Tags: OpenWRT, IPv6, DHCPv6, nftables, home lab, AI tooling, networking, Proxmox

Running LLMs Locally: AMD APU vs Discrete GPU — Why Architecture Matters More Than Hardware

The Hardware

I benchmarked two very different local AI setups:

Matt-Mini — a Windows Mini PC that most people would dismiss for AI:
- CPU: AMD Ryzen 7 5800U (8 cores, Zen 3)
- iGPU: AMD Radeon Vega 8 (integrated, shared memory)
- RAM: 64GB DDR4-3200 (~50 GB/s bandwidth)

Ubuntu Laptop — a more conventional AI workstation:
- GPU: NVIDIA RTX 4070 8GB VRAM (~300 GB/s GDDR6X bandwidth)
- RAM: DDR5 system RAM (~80–100 GB/s), separate from GPU VRAM

The critical insight about the APU: the iGPU uses shared system memory as VRAM. With 64GB of RAM, the GPU can access tens of gigabytes for model weights — something impossible on a discrete GPU with fixed VRAM. The trade-off is bandwidth: DDR4 gives ~50 GB/s vs the RTX 4070's ~300 GB/s.

---

The Benchmark Setup

I used Ollama as the inference server (Vulkan backend for AMD iGPU — no ROCm required) and ran three prompts per model:

• Short: "What is 2 + 2? Answer in one word." — tests base throughput
• Reasoning: A multi-step maths problem — tests sustained generation
• Coding: Fibonacci with memoization in Python — tests structured output

Metric: tokens per second (TPS) for generation.

---

Results: Matt-Mini (AMD Ryzen 7 5800U + Vega 8 iGPU, 64GB shared RAM)

Model Architecture Comparison (all Q4_K_M)

Model	Avg TPS	Total Params	Active Params	Type
qwen3:30b-a3b	12.0	30B	3B	MoE
qwen3-coder:30b-a3b	12.1	30B	3B	MoE (coding)
qwen3:8b	5.3	8B	8B	Dense
qwen3.5-abliterated:35b-a3b	4.65	35B	~3.5B	MoE (uncensored)
qwen3.5-opus-distill	3.83	35B	~3.5B	MoE (distilled, Q8_0)
mixtral:8x7b	3.5	46.7B	12.9B	MoE
deepseek-r1:14b	3.1	14B	14B	Dense

Q4_K_M vs Q8_0 on Bandwidth-Constrained iGPU

The Vega 8 iGPU is bottlenecked by DDR4 memory bandwidth (~50 GB/s). Q8_0 uses 2× the memory bandwidth of Q4_K_M with no compute benefit on hardware lacking AVX_VNNI. The speed penalty is significant:

Model	Q4_K_M TPS	Q8_0 TPS	Q4 faster by
qwen3-coder:30b-a3b	12.1	7.73	+57%
qwen3.5-abliterated:35b-a3b	4.65	3.83	+21%

Use Q4_K_M on the APU. Q8_0 only makes sense if quality is paramount and you can accept the speed penalty.

---

Results: Ubuntu Laptop (NVIDIA RTX 4070 8GB, DDR5)

General and Reasoning Models

Model	Avg TPS	Params	Notes
qwen2.5-coder:1.5b	163	1.5B	Tiny, saturates GPU
qwen2.5-coder:7b	52	7B	Fast in VRAM
qwen3.5:4b	51	4B
deepseek-r1:7b	39	7B	Strong reasoning, consistent TPS
qwen3-vl:8b	35	8B	Vision model
llama3.1:latest	36	8B
qwen3.5:latest	24	~14B	Starts hitting VRAM limit
qwen3.5:27b	3.0	27B	Exceeds 8GB VRAM, spills to RAM

Vision Models (for ComfyUI and multimodal workflows)

Model	Avg TPS	VRAM	Notes
qwen3-vl:4b-instruct-q8_0	45	~5.5GB	Best balance — fast, high quality, leaves headroom
qwen3-vl:8b-instruct-q4_K_M	35	~5.5GB	Larger model, slightly slower, better comprehension
minicpm-v:8b-2.6-q4_K_M	38	~5GB	Fast but terse — short responses on text tasks
qwen2.5vl:3b-q8_0	15	~3.5GB	Slow despite small size — VRAM load overhead

The dramatic drop from qwen3.5:latest (~24 TPS) to qwen3.5:27b (3 TPS) marks the VRAM cliff. Once the model no longer fits in 8GB, it spills to system RAM — but even though this machine has fast DDR5, the bottleneck becomes the PCIe bus (~32 GB/s) between the GPU and system memory, not the RAM speed itself. Performance collapses to APU-level speeds despite the faster RAM.

---

The Key Finding: Active Parameters Are What Matter

The headline result is qwen3:30b-a3b hitting 12 TPS — faster than the 8B dense model, despite having 30 billion total parameters.

This seems counterintuitive until you understand Mixture of Experts (MoE) architecture. In a MoE model, the network is split into many "expert" sub-networks. For any given token, only a small subset of experts are activated. qwen3:30b-a3b has 30B total parameters but only 3B active per token — the same compute cost per token as a 3B dense model, but with the knowledge capacity of a 30B model.

The rule that emerges from these results:

MoE speed advantage only materialises when active parameter count is kept low.

Look at mixtral:8x7b: it's MoE, but with 12.9B active parameters per token. Despite the MoE structure it runs at the same speed as the dense 14B model — because the active compute is similar.

qwen3:30b-a3b wins because it keeps active params at just 3B while maximising total capacity.

---

The Two Hardware Stories

Discrete GPU: Fast but VRAM-limited

The RTX 4070 hits 35–163 TPS for models that fit in 8GB VRAM. It's fast — bandwidth is not the bottleneck. But the moment a model exceeds 8GB, performance falls off a cliff: qwen3.5:27b drops to 3 TPS, identical to the APU. The discrete GPU is a sprinter with a hard wall.

Shared-Memory APU: Slow but capacious

The Vega 8 iGPU runs at 3–12 TPS — slower across the board for models that fit in discrete VRAM. But it can run a 34GB Q8_0 model that would never fit on the RTX 4070. The APU is a distance runner with no wall.

Where they meet

When a model exceeds the discrete GPU's VRAM, both machines run at the same ~3 TPS. At that point, the APU's 64GB capacity advantage becomes the deciding factor — it can run larger models at equal speed, with Q8_0 quality instead of being forced into aggressive quantization.

The MoE Sweet Spot for APUs

Low active-parameter MoE is the ideal architecture for shared-memory systems: fewer active params = less bandwidth per token = more TPS on bandwidth-constrained DDR4. qwen3:30b-a3b at 12 TPS demonstrates this perfectly — 30B total parameters, but only 3B active, running faster than the dense 8B model.

---

Practical Recommendations

For AMD APU systems with 32GB+ unified memory (Ryzen 5800U, no AVX_VNNI):
1. Use qwen3:30b-a3b or qwen3-coder:30b-a3b as your default — ~12 TPS, best speed/quality
2. Use Q4_K_M, not Q8_0 — Q8_0 is 20–57% slower on bandwidth-limited DDR4; AVX_VNNI (which would offset the bandwidth cost) is not present on Zen 3
3. Prefer MoE models with low active param counts (under 4B active) — this is the single biggest performance lever
4. Ollama with Vulkan is the easiest path — no ROCm build required, works out of the box
5. Disable sleep — large model downloads will resume but you waste time

For discrete GPU systems (e.g. RTX 4070 8GB, Intel Ultra 7 165H with AVX_VNNI):
1. Match model size to VRAM — keep total model size under ~7.5GB to stay fully in VRAM
2. Q4_K_M for 7–8B models at this VRAM level — fits comfortably with headroom
3. Q8_0 is viable for vision models under 6GB (e.g. qwen3-vl:4b-instruct-q8_0) — AVX_VNNI on the host CPU means Q8_0 CPU fallback is no slower
4. For ComfyUI inpainting: qwen3-vl:4b-instruct-q8_0 at 45 TPS uses ~5.5GB, leaving room for the diffusion model
5. Avoid models that spill to RAM — PCIe bandwidth (~32 GB/s) becomes the bottleneck, not DDR5
6. For larger models, the APU is a natural complement — it runs 30B+ at equal speed to any spilling model

---

Tools Used

• Ollama — inference server, Vulkan backend
• llmfit — hardware-fit recommender (useful for finding candidate models, but note: speed estimates for Vega 8 iGPU are inaccurate — it assumes 180 GB/s ROCm bandwidth vs the real ~50 GB/s)
• benchmark_ollama.py — custom benchmark script measuring TPS across models and prompt types

---

Tested April 2026 on Ollama — AMD Ryzen 7 5800U (Vega 8 iGPU, 64GB DDR4) and NVIDIA RTX 4070 8GB (DDR5 system RAM).

LiteLLM + Agent Teams: A Practical Guide

An aide memoire for using the local AI infrastructure day-to-day.

---

The big picture

You have three layers:

Your task (plain English)
        ↓
  Agent team (Python, OpenAI Agents SDK)
        ↓
  LiteLLM proxy  ←→  Ollama (local GPU)
                 ←→  OpenRouter (cloud free)
                 ←→  Anthropic (claude-haiku)

LiteLLM is a translation layer. It gives everything a single OpenAI-compatible URL (http://10.140.20.63:4000/v1) regardless of whether the model is running locally on your GPU or fetched from a cloud provider. Your code never changes — only the model name string changes.

The agent team is a set of specialised AI workers. You give the orchestrator a task in plain English; it decides which specialist to hand it to; the specialist does the work and hands results back.

---

Part 1 — Using LiteLLM directly

From the command line (curl)

# Ask any model a question
curl http://10.140.20.63:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key-needed" \
  -d '{
    "model": "qwen3.5:4b",
    "messages": [{"role": "user", "content": "What is a BGP route reflector?"}]
  }'

# List all available models
curl http://10.140.20.63:4000/v1/models | python3 -m json.tool | grep '"id"'

From Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://10.140.20.63:4000/v1",
    api_key="no-key-needed",
)

response = client.chat.completions.create(
    model="qwen3.5:4b",   # or "claude-haiku-4-5", "nemotron-120b", etc.
    messages=[{"role": "user", "content": "Summarise this log: ..."}],
)
print(response.choices[0].message.content)

Choosing a model

Use case	Model string	Where it runs
Quick questions, triage	qwen3.5:4b	Local GPU (3.4 GB)
Writing code	qwen2.5-coder:7b	Local GPU (4.7 GB)
General analysis	qwen3.5	Local GPU (6.6 GB)
Images / screenshots	qwen3-vl	Local GPU (6.1 GB)
Heavy reasoning	nemotron-120b	Cloud free (OpenRouter)
Reliable tool calling	claude-haiku-4-5	Cloud (Anthropic/OpenRouter)
Best available free	free	Cloud free (auto-routed)

Group aliases — if the specific model is busy or unavailable, LiteLLM falls back automatically:

Alias	Primary	Fallback
fast	qwen3.5:4b	qwen2.5-coder:1.5b
coder	qwen2.5-coder:7b	qwen2.5-coder:1.5b
local	qwen3.5	llama3.1
reasoning	nemotron-120b	gpt-oss-120b

Health check

curl http://10.140.20.63:4000/health
incus exec litellm -- journalctl -u litellm -f   # live logs

---

Part 2 — Running the agent team

The one-liner

cd /home/user/claude/agents
.venv/bin/python team.py "your task here"

Example tasks

# Coding
.venv/bin/python team.py "write a Python script that tails a log file and alerts on ERROR lines"

# Research
.venv/bin/python team.py "what are the main CVEs in OpenSSH versions 8.x to 9.x?"

# Analysis
.venv/bin/python team.py "analyse this nmap output and prioritise the findings: [paste output]"

# Mixed — the orchestrator chains specialists automatically
.venv/bin/python team.py "research the log4shell vulnerability then write a Python checker for it"

What happens under the hood

You: "research log4shell then write a checker"
        ↓
Orchestrator (claude-haiku) reads task
        ↓
Handoff → Researcher (nemotron-120b, cloud)
  "Log4Shell is CVE-2021-44228, affects Log4j 2.0–2.14.1..."
        ↓
Back to Orchestrator → Handoff → Coder (qwen2.5-coder:7b, local GPU)
  "def check_log4shell(host, port): ..."
        ↓
Orchestrator summarises and returns to you

The orchestrator uses haiku because it reliably produces valid tool-call JSON for handoffs. Local Ollama models are fast but unreliable at structured function-calling.

Watching it work

Add LITELLM_LOG=DEBUG to see every model call:

LITELLM_LOG=DEBUG .venv/bin/python team.py "hello"

Or watch the LiteLLM proxy logs live in another terminal:

incus exec litellm -- journalctl -u litellm -f

---

Part 3 — Writing your own agents

Minimal single agent

import asyncio, os
os.environ["OPENAI_BASE_URL"] = "http://10.140.20.63:4000/v1"
os.environ["OPENAI_API_KEY"]  = "no-key-needed"

from agents import Agent, Runner

agent = Agent(
    name="Helper",
    model="qwen3.5:4b",
    instructions="You are a helpful assistant. Be concise.",
)

async def main():
    result = await Runner.run(agent, "What is ARP spoofing?")
    print(result.final_output)

asyncio.run(main())

Adding tools (things agents can do)

from agents import Agent, Runner, function_tool
import httpx

@function_tool
async def get_url(url: str) -> str:
    """Fetch the contents of a URL."""
    async with httpx.AsyncClient(timeout=10) as c:
        r = await c.get(url)
        return r.text[:2000]   # truncate to avoid context overflow

agent = Agent(
    name="WebReader",
    model="qwen3.5:4b",
    instructions="You can fetch URLs to answer questions.",
    tools=[get_url],
)

Rule: tools are Python functions decorated with @function_tool. The agent decides when to call them. The docstring becomes the tool description — make it clear.

Handing off between agents

from agents import Agent, Runner, handoff

specialist = Agent(
    name="Specialist",
    model="qwen3.5",
    instructions="You handle detailed analysis. Return results clearly.",
)

orchestrator = Agent(
    name="Orchestrator",
    model="claude-haiku-4-5",
    instructions="Route analysis tasks to Specialist. Summarise results.",
    handoffs=[handoff(specialist)],
)

result = await Runner.run(orchestrator, "Analyse this data: ...")

handoff() is itself a tool the orchestrator can call. When it calls it, execution transfers to the specialist; when the specialist finishes, control returns to the orchestrator.

The existing tools you can reuse

gpu_tools.py — for any agent that needs to know about the GPU:

from gpu_tools import vram_status, list_local_models, comfyui_status
agent = Agent(..., tools=[vram_status, list_local_models])

devops_tools.py — for agents that manage containers:

from devops_tools import container_run, container_write_file, container_read_file, http_probe, container_systemctl
agent = Agent(..., tools=[container_run, http_probe])

---

Part 4 — Practical patterns

Pattern 1: Quick one-shot query

Use make_client() from litellm_client.py directly — no agent overhead:

from litellm_client import make_client, FAST_MODEL

async def ask(question: str) -> str:
    client = make_client()
    resp = await client.chat.completions.create(
        model=FAST_MODEL,
        messages=[{"role": "user", "content": question}],
    )
    return resp.choices[0].message.content

Pattern 2: Task with a deadline / retry limit

result = await Runner.run(agent, task, max_turns=10)

max_turns prevents infinite loops. The team.py orchestrator uses 40 turns because research+code tasks can take many steps.

Pattern 3: Streaming output

from agents import Runner

async for event in Runner.run_streamed(agent, task):
    if hasattr(event, "delta") and event.delta:
        print(event.delta, end="", flush=True)

Pattern 4: DevOps / automation agent

See setup_tts_stt.py as a reference. The pattern is:
1. Write a detailed task string explaining exactly what the agent should do and verify
2. Give it the right tools (container_run, http_probe, etc.)
3. Set instructions to "act immediately, don't ask permission"
4. Set max_turns=40 for multi-step work

agent = Agent(
    name="DevOps",
    model="claude-haiku-4-5",   # must use haiku — local models can't do tool-calling
    tools=[container_run, container_write_file, http_probe, container_systemctl],
    instructions="Act immediately. Never ask for permission. Verify each step.",
)
result = await Runner.run(agent, TASK, max_turns=40)

---

Part 5 — Gotchas and tips

Local models can't do structured tool-calling

qwen3.5, qwen2.5-coder:7b, etc. produce good prose but often garble the JSON format needed for handoff() and @function_tool calls. Always use claude-haiku-4-5 as your orchestrator — it's reliable and cheap (Anthropic free tier via OpenRouter).

Only one large model fits in VRAM at a time

The RTX 4070 has 8 GB. If you ask the orchestrator to hand off to a 6.6 GB local model while another 4.7 GB model is loaded, Ollama unloads the first one. There is a ~5–15 second cold-load delay. This is normal.

Free cloud models are rate-limited

nemotron-120b and other OpenRouter free models may queue or time out under load. If an agent stalls for >2 minutes with no output, it's usually rate-limiting. Switch to gpt-oss-120b or qwen3-80b as alternatives.

The free model alias changes

openrouter/openrouter/free routes to whatever OpenRouter considers the best free model at that moment. Good for exploration; use a specific model name for reproducible pipelines.

Ollama keep-alive

Models stay in VRAM for 15 minutes after last use (KEEP_ALIVE=15m). If you want to free VRAM immediately:

curl -X POST http://10.140.20.1:11434/api/generate -d '{"model":"qwen3.5","keep_alive":0}'

---

Part 6 — Agent Team in Open WebUI

The agent team is exposed as a model in Open WebUI via the Pipelines server — a small FastAPI app that sits between Open WebUI and the agent code.

Open WebUI chat
      ↓  (selects "Agent Team" model)
Pipelines server  (host: 10.140.20.1:9099)
      ↓
Agent orchestrator (claude-haiku)
      ↓  handoffs
Specialist agents (local GPU / cloud free)

Architecture files

File	Purpose
agents/pipelines/agent_team.py	The pipeline class — wraps the agent team
agents/run_pipelines.sh	Manual start script
/etc/systemd/system/owui-pipelines.service	Systemd service (starts on boot)

Managing the pipelines server

sudo systemctl status owui-pipelines
sudo systemctl restart owui-pipelines
sudo journalctl -u owui-pipelines -f

Connecting to Open WebUI (one-time setup)

1. Open http://localhost:3001
2. Top-right avatar → Admin Panel
3. Settings → Connections → Pipelines
4. Add:
5. URL: http://10.140.20.1:9099
6. API Key: 0p3n-w3bu!
7. Click Save — "Agent Team" now appears in the model picker

Using it

Select Agent Team in the model picker and chat normally. Each message is routed by the orchestrator to the right specialist. The full conversation history is passed so the team has context across turns.

The pipelines server API key (0p3n-w3bu!) is the default from the open-webui-pipelines package. Change it in /etc/systemd/system/owui-pipelines.service and update the Open WebUI connection setting to match.

Adding more pipelines

Drop a new .py file with a Pipeline class into agents/pipelines/, then:

sudo systemctl restart owui-pipelines

The new pipeline appears as a model in Open WebUI immediately.

---

Quick reference card

# Run agent team
cd /home/user/claude/agents && .venv/bin/python team.py "task"

# Query a model directly
curl http://10.140.20.63:4000/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer no-key-needed" \
  -d '{"model":"qwen3.5:4b","messages":[{"role":"user","content":"hello"}]}'

# List models
curl -s http://10.140.20.63:4000/v1/models | python3 -m json.tool | grep '"id"'

# Watch LiteLLM traffic
incus exec litellm -- journalctl -u litellm -f

# Check VRAM
curl -s http://10.140.20.1:11434/api/ps | python3 -m json.tool

# Add a model to Ollama
ollama pull <model-name>
# Then add it to /etc/litellm/config.yaml and push + restart

---

File map

/home/user/claude/agents/
├── team.py            ← entry point — run this
├── litellm_client.py  ← model constants and URLs
├── gpu_tools.py       ← tools: vram_status, list_local_models, comfyui_status
├── devops_tools.py    ← tools: container_run, container_write_file, http_probe, ...
├── setup_tts_stt.py   ← reference: single-purpose DevOps agent
└── .venv/             ← virtualenv (openai-agents, openai)

/etc/litellm/
├── config.yaml        ← model list (edit on host, push to container)
└── secrets.env        ← OPENROUTER_API_KEY

CPU vs. GPU: Is Hardware Acceleration Always Faster for Real-Time TTS?

Following up on my last post about fixing progressive streaming in Kokoro FastAPI, I decided to take things a step further. If the goal is minimizing latency for a conversational AI assistant, shouldn't throwing a dedicated GPU at the problem make it even faster?

I spent the afternoon duplicating my streaming container and configuring it to run on a local NVIDIA GeForce RTX 4070 (8GB). The results were... surprising. It turns out that for real-time, sentence-by-sentence streaming, "faster" hardware doesn't always translate to a better user experience.

---

The Setup: Moving to Incus and CUDA

While my previous tests were in Podman, I've recently moved to Incus for better resource management. I duplicated the kokoro-stream container to a new sandbox named kokoro-stream-gpu and passed through the GPU:

incus config device add kokoro-stream-gpu mygpu gpu uid=1000 gid=1000
incus config set kokoro-stream-gpu nvidia.runtime true
incus config set kokoro-stream-gpu nvidia.driver.capabilities compute,utility,video

Inside the container, I switched the backend from the ONNX CPU runtime to the PyTorch GPU version. I also had to port over the same split_pattern and asyncio.sleep(0) fixes from the last session to ensure I was comparing apples to apples (sentence-level streaming vs. sentence-level streaming).

---

The Benchmark: Short vs. Long Form

I ran two tests using the British English male voice (bm_fable): one with a short two-sentence phrase (~90 chars) and one with the full text of my last blog post (~8,700 chars).

Metric	CPU (ONNX)	GPU (RTX 4070)	Speedup
TTFA (Short Text)	~557 ms	~508 ms	1.1x
Total Time (Long Text)	~289 s	~15 s	19.2x
Throughput (Long Text)	~30 char/s	~580 char/s	19.2x
System RAM Usage	1.21 GiB	1.92 GiB	-
Video RAM (VRAM)	0 MB	~850 MB	-

---

Reflections: When is the GPU worth it?

The results tell two very different stories depending on what you're doing.

1. Conversational AI (Short Sentences)

If you're building a real-time voice assistant that speaks one or two sentences at a time, the CPU is the clear winner. The Time to First Audio (TTFA) is virtually identical because the overhead of initializing the GPU pipeline eats up any compute gains. For this use case, the GPU is just an expensive way to use more RAM.

2. Long-Form Content (Articles, Blog Posts)

This is where the RTX 4070 absolutely screams. When I threw the full 8,700-character blog post at it, the GPU version finished the entire synthesis in 15 seconds. The CPU version was still grinding away at nearly the 5-minute mark.

At 580 characters per second, the GPU isn't just "faster"—it changes the nature of the service. You can listen to an entire article almost as soon as you click "Generate."

The Verdict

• Stick with CPU for: Open WebUI, chatbots, home assistants, and low-RAM servers.
• Switch to GPU for: Audiobook generation, long-form reading, or high-concurrency environments.

The kokoro-stream-gpu container is now my go-to for "reading" long documentation, while the CPU version remains my daily driver for conversational chat.

---

The Evidence: Benchmarking Code

To keep things evidence-based, here is the Python script used to capture these metrics. It probes the streaming API and measures exactly when the first and last chunks arrive.

1. Throughput & Latency Probe (benchmark_long.py)

import time
import requests
import subprocess

# Ports
GPU_URL = "http://localhost:8881/v1/audio/speech"
CPU_URL = "http://localhost:8882/v1/audio/speech"

# Load long text
with open("blog_post.md", "r") as f:
    LONG_TEXT = f.read()

def run_benchmark(name, url):
    print(f"\n--- Benchmarking {name} ---")
    start_time = time.time()
    first_chunk_time = None

    payload = {
        "input": LONG_TEXT,
        "voice": "bm_fable",
        "response_format": "mp3",
        "stream": True
    }

    with requests.post(url, json=payload, stream=True) as r:
        r.raise_for_status()
        for chunk in r.iter_content(chunk_size=1024):
            if chunk and first_chunk_time is None:
                first_chunk_time = time.time() - start_time

        total_time = time.time() - start_time

    return {
        "ttfa_ms": round(first_chunk_time * 1000, 2),
        "total_s": round(total_time, 2),
        "char_s": round(len(LONG_TEXT) / total_time, 2)
    }

2. Evidence Audio Generation (generate_evidence.py)

import requests
import hashlib

def generate_and_hash(url, filename):
    r = requests.post(url, json={"input": LONG_TEXT, "voice": "bm_fable"})
    with open(filename, "wb") as f:
        f.write(r.content)
    return hashlib.md5(r.content).hexdigest()

# Results:
# CPU Hash: a22fe5e4d70a2888d755e0f8df7dae8f
# GPU Hash: e5ccba5c22ef3edf594aabaa2c08bb5f

Running Incus on Ubuntu 24.04. Hardware: NVIDIA GeForce RTX 4070 8GB. Frameworks: ONNX Runtime (CPU) vs. PyTorch 2.6+CUDA 12.4 (GPU).

Making Local TTS Actually Stream: Fixing Kokoro FastAPI for Real-Time Audio

Making Local TTS Stream: Fixing Kokoro FastAPI for Real-Time Audio

If you've been following along with my local AI setup, you'll know I run most of my services in Podman containers on a home server — Ollama, Open WebUI, FasterWhisper, and a handful of other tools.  One of those is Kokoro FastAPI, a self-hosted text-to-speech server based on the Kokoro-82M ONNX model.  It produces surprisingly good speech, supports multiple voices and languages, and exposes an OpenAI-compatible endpoint compatible with Open WebUI.

This post covers a productive session, it makes a change, where what started as a simple Firefox bug turned into a full streaming pipeline investigation — with benchmarks, a duplicate container sandbox, and a fix that meaningfully reduces time-to-first-audio for conversational use cases.

---

Firefox MediaSource Bug

First thing first: the web UI at kokoro-web.lan worked fine in Chrome but threw this in Firefox when you clicked Generate Speech:

MediaSource.addSourceBuffer: Type not supported in MediaSource

The culprit was this single line in AudioService.js:

this.sourceBuffer = this.mediaSource.addSourceBuffer('audio/mpeg');

Firefox does not support audio/mpeg in the MediaSource Extensions (MSE) API.  Chrome does.  Why not just use Chrome?  Well, I simply prefer Firefox...  The fix was to check for support in the script first, and fall back to a simpler approach when MSE isn't available:

if (!window.MediaSource || !MediaSource.isTypeSupported('audio/mpeg')) {
    await this.setupBufferedStream(stream, response, onProgress, estimatedChunks);
    return;
}

The setupBufferedStream fallback collects all incoming audio chunks into a Blob and sets it as a plain audio.src — no MSE required, works everywhere. The patched file is saved locally and injected via podman cp rather than rebuilding the image.

---

Benchmarking: Does Format or Voice Matter?

With the Firefox issue sorted, I ran latency benchmarks across the three supported output formats and three voices, using a consistent test phrase:

"I love Mediclinic, but I think there is a lot of scope for the EHR development to go awry."

Three runs per combination, stream: false, measured with Python's time.perf_counter().

By format (averaged across all voices)

Format	Avg latency	File size
WAV	1382 ms	~256 KB
PCM	1417 ms	~256 KB
MP3	1457 ms	~86 KB

By voice (averaged across all formats)

Voice	Description	Avg latency
af_heart	American English female	1379 ms
bm_fable	British English male	1439 ms
ef_dora	Dutch female	1438 ms

The takeaway: format and voice choice barely matter for latency. The ONNX inference dominates — everything else (MP3 encoding, voice model differences) contributes at most ~80 ms.  MP3 is still the right default for web playback given its file size advantage. The Dutch voice (ef_dora) performs on par with the English voices, which is a good sign for multilingual deployments.

---

When stream: true, doesn't stream

The Kokoro API has a stream: true parameter. For a conversational application, this should mean the server sends the first sentence's audio while it's still generating the second — reducing perceived latency significantly. I modified the test phrase to have two clear sentences:

"I love Mediclinic. But I think there is a lot of scope for the EHR development to go awry."

Then I wrote a Python probe to track exactly when each 1 KB chunk arrived at the client:

t_start = time.perf_counter()
chunks = []
with urllib.request.urlopen(req) as resp:
    while True:
        chunk = resp.read(1024)
        if not chunk: break
        t = round((time.perf_counter() - t_start) * 1000)
        chunks.append((t, len(chunk)))

print(f"First chunk: {chunks[0][0]}ms")
print(f"Last chunk:  {chunks[-1][0]}ms")

Results for stream: true, af_heart, MP3:

First chunk: 1462ms
Last chunk:  1464ms
Chunks: 89

All 89 chunks arrived within 2 ms of each other, after a full 1.4 second wait. stream: false was identical. Even PCM format — which has zero encoder overhead — showed the same pattern. Something was buffering the entire audio before sending a single byte.

---

Chunks all arriving at once is not streaming

I spun up a duplicate container, kokoro-stream, on port 8881 as an isolated sandbox, and set about tracing the pipeline.  The server code is actually well-architected: async generators and yield statements all the way from the HTTP handler down to the ONNX inference layer. The StreamingResponse even sets X-Accel-Buffering: no. On paper, it should stream.

I identified three hypotheses:

	Hypothesis	Evidence for
H1	ONNX inference batches both sentences as one call	PCM (no encoder) also shows simultaneous delivery
H2	Uvicorn buffers the response body below a threshold	No asyncio yield points between sentence yields
H3	PyAV MP3 encoder buffers early frames	Secondary — can't explain PCM behaviour

What the code actually does

Inside tts_service.py, smart_split() splits the input text into chunks before inference — good. But it batches sentences together when their combined token count is under 250 tokens. The two-sentence test input is only 105 tokens, so both sentences were delivered as a single string to KokoroV1.generate().

Inside kokoro_v1.py, the pipeline was called with split_pattern=r'\n+' — meaning it would only split on newlines. Since there were no newlines, both sentences went through a single ONNX inference call and produced a single audio yield. No amount of async wiring downstream could fix that.

Even if the sentences had been processed separately, the for result in pipeline(...) loop is synchronous — it never returns control to the asyncio event loop between sentences, so the HTTP layer has no opportunity to flush.

The fix

Two minimal changes to kokoro-stream only:

inference/kokoro_v1.py — change the pipeline split pattern to break on sentence-ending punctuation:

# before
split_pattern=r'\n+'
# after
split_pattern=r'(?<=[.!?])\s+'

inference/kokoro_v1.py and services/tts_service.py — add asyncio yield points between sentence yields:

yield AudioChunk(...)
await asyncio.sleep(0)  # return control to event loop → HTTP layer can flush

Before and after

Metric	Before	After
First chunk (TTFA)	~1400 ms	~575 ms
Last chunk	~1400 ms	~1400 ms
Gap	~2 ms	~1100 ms

First sentence audio now arrives at the client at ~575 ms while the second sentence is still being synthesised. Total generation time is unchanged — we're not making the model faster, we're just not making the user wait for everything before delivering anything.

---

Setup

Both containers are now accessible via .lan hostnames using Caddy as a reverse proxy:

URL	Container	Port	Notes
https://kokoro-web.lan	kokoro-tts	8880	Production
https://kokoro-stream.lan	kokoro-stream	8881	Streaming-optimised

Open WebUI is configured to use the production container at port 8880. The streaming container is available for direct use and API calls where lower TTFA matters.

---

Conclusions

A few things worth noting from this session:

The architecture was already correct. The Kokoro FastAPI codebase uses async generators properly throughout — the issue wasn't bad design, it was two small configuration defaults that compounded badly for short inputs. The token batching threshold (250 tokens) and the newline-only split pattern made sense in isolation but combined to eliminate sentence-level streaming entirely for typical conversational inputs.

PCM as a diagnostic tool. Benchmarking PCM format (raw samples, no encoding) alongside MP3 was valuable precisely because it let us eliminate the audio encoder as a suspect early. When PCM and MP3 showed identical behaviour, we knew the bottleneck was upstream of the encoder.

asyncio.sleep(0) is surprisingly powerful. A zero-duration sleep doesn't actually sleep — it just yields control back to the event loop. That's enough to let uvicorn flush pending response bytes to the socket. It's a one-line fix with a meaningful impact on perceived latency.

The full benchmark data, pipeline analysis, and change logs are all documented if you want to replicate this setup.

---

Running Podman on Ubuntu 24.04. Kokoro FastAPI image: ghcr.io/remsky/kokoro-fastapi-cpu:latest. Voices used: af_heart, bm_fable, ef_dora.

cuda-graphs-vllm-gdn-hybrid-qwen35-9b

title: "Getting CUDA Graphs Working on vLLM with a GDN Hybrid Model (Qwen3.5-9B)"
date: 2026-06-26

---

Getting CUDA Graphs Working on vLLM with a GDN Hybrid Model (Qwen3.5-9B)

I've been running Qwen3.5-9B-AWQ on a Xenon box with an RTX 4060 Ti 16 GB via vLLM 0.20.2. Decode throughput was sitting at around 7–8 tok/s on single-request agentic use, which felt wrong — the 4060 Ti has 288 GB/s of VRAM bandwidth and a 9B AWQ model should be doing considerably better than that. The research notes from May pointed at --enforce-eager as the culprit (CUDA graph capture disabled, 5–10× penalty) but also noted it was required at 239K context for VRAM headroom. That was where things sat for a few weeks.

Today I did the actual sweep to find out what's achievable — and the answer turned out to be more nuanced than "remove the flag and you're done".

Some background on the architecture

Qwen3.5-9B is a GDN (GatedDeltaNet) hybrid, not a pure dense transformer. It has 32 layers: 8 standard attention layers (KV-cached, accelerated by AWQ Marlin) and 24 linear attention / SSM recurrent layers (no KV cache, run unconditionally every token using the FLA kernel). The SSM layers are why the model behaves differently to a vanilla 9B — they eat into the VRAM budget in ways the standard vLLM probe logic doesn't account for, and they're also why --enforce-eager is needed at high context: the FLA activation tensor grows with sequence length (~1 KB/token) and at GMU=0.98 you simply don't have the headroom for CUDA graph capture on top of that.

One important flag note: do not use ngram speculative decoding on GDN hybrids — SSM state rollback on rejected speculative tokens corrupts the recurrent state (vLLM issues #39273 and #40875). MTP speculative decoding is safe because the draft heads are model-native.

The sweep

I ran five configurations, each benchmarked with three consecutive requests of 250 tokens and wall-clock timing. Run 1 is always slower (JIT/FLA kernel warmup on first request after startup), so I quote the median of runs 2–3:

Config	max_model_len	enforce-eager	MTP	CUDA graphs	tok/s
Baseline	239,616	Yes	No	No	7.75
Tier 1	65,536	No	Yes (n=2)	Yes	31.5
Tier 2 ★	108,768	No	Yes (n=2)	Yes	31.3
Tier 3	168,000	Yes	Yes (n=2)	No	12.6
Tier 4	239,616	Yes	No	No	8.0

The key insight from Tier 1 vs Tier 2: both give identical throughput. CUDA graphs run at the same fixed capture sizes ([1, 2, 4, 8, 16, 24, 32, 40, 48] tokens) regardless of max context, so 108K is strictly better than 65K for free. That's the config I've settled on.

The MTP VRAM surprise

Adding --speculative-config '{"method":"mtp","num_speculative_tokens":2}' loads an MTP draft head that borrows the base model's embedding and lm_head weights but adds its own additional layer — costing roughly 0.5 GiB of VRAM. This is not reflected in the existing context ceiling probe table (which was built without MTP), so the real ceiling with MTP is lower than the probe table suggests.

Concretely: at GMU=0.90, the original probe gave 80K tokens without MTP. With MTP, the available KV cache at GMU=0.90 drops below what 65K requires (1.36 GiB needed, 1.02 GiB available). I had to nudge to GMU=0.93 to get 65K working, and GMU=0.95 for 108K. The updated ceiling table:

GMU	enforce-eager	MTP	Max context
0.90	No	Yes	~46K (OOM)
0.93	No	Yes	65,536 ✓
0.95	No	Yes	108,768 ✓
0.97	Yes	Yes	~170,000
0.99	Yes	No	239,616 ✓

The 200K+ context tiers I'd hoped to test with MTP turned out to be impossible on 16 GB — reaching 200K with MTP + enforce-eager would require more VRAM than even GMU=1.0 can provide. If you want 200K+ you have to drop MTP (Tier 4 above, 8.0 tok/s) or use a different backend entirely.

MTP does help even on enforce-eager configs though: compare Tier 3 (168K, eager, MTP) at 12.6 tok/s versus Tier 4 (239K, eager, no MTP) at 8.0 tok/s — the draft head adds about 1.6× at the cost of ~70K of context headroom.

First-startup compile time

With --enforce-eager removed, vLLM runs torch.compile (Inductor) and CUDA graph capture on startup. The first cold start for the backbone takes about 130 seconds; the MTP eagle head takes another 40 seconds on top of that. Subsequent startups read from /root/.cache/vllm/torch_compile_cache/ and complete in about 16 seconds. The cache key includes GMU, so changing that value invalidates it and you pay the full compile time again.

Tier 2 also crashes on its first boot attempt and recovers on the auto-restart (a timing issue in the profiling phase, I suspect). With Restart=on-failure and RestartSec=30 in the service file, it comes up cleanly on the second try — a bit inelegant but consistent.

The systemd ExecStart gotcha

If you're writing the service file programmatically, be careful with the speculative-config JSON. Double-quoted backslash-escaped JSON in ExecStart:

ExecStart=... --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":2}"

...gets silently stripped by systemd to {method:mtp,num_speculative_tokens:2}, which is not valid JSON and fails with Value cannot be converted to <function loads>. Single-quoted JSON works correctly:

ExecStart=... --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

I ended up writing service files with Python's open(...).write(...) to avoid any heredoc quoting nonsense:

MTP = "'" + '{"method":"mtp","num_speculative_tokens":2}' + "'"
# then embed MTP in the ExecStart f-string

Production config (as of 2026-06-26)

# /etc/systemd/system/vllm.service inside LXC 8003 on Xenon (RTX 4060 Ti 16 GB)
[Service]
Environment=CUDA_VISIBLE_DEVICES=0
Environment=VLLM_WORKER_MULTIPROC_METHOD=spawn
Environment=PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
Environment=VLLM_SLEEP_WHEN_IDLE=1
ExecStart=/opt/vllm-env/bin/vllm serve /mnt/models/Qwen3.5-9B-AWQ \
  --host 0.0.0.0 --port 8000 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 \
  --max-model-len 108768 \
  --enable-prefix-caching \
  --language-model-only \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --max-num-seqs 8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
Restart=on-failure
RestartSec=30
TimeoutStartSec=600

31 tok/s at 108K context, CUDA graphs and MTP both active, ~15.1 GB VRAM used at rest. That's a four-fold improvement over the --enforce-eager baseline for agentic single-stream use.

If you need context beyond 108K, the choices are: Tier 3 (168K, 12.6 tok/s, MTP on, enforce-eager) or Tier 4 (239K, 8.0 tok/s, MTP off, enforce-eager). Neither is as fast as Tier 2, but at least now there's a table rather than a guess.

References

• vLLM v0.20.2 — OpenAI-compatible inference engine for LLMs; issues #39273 and #40875 cover ngram + SSM state corruption
• Qwen3.5-9B-AWQ — Alibaba's 9B GDN hybrid model, AWQ-quantised; 32 layers (8 attention + 24 SSM/linear attention)
• FLA (Flash Linear Attention) — CUDA kernels for GDN/GatedDeltaNet linear attention layers; the activation buffer grows with sequence length
• torch.compile — PyTorch graph compilation; Inductor backend used by vLLM's CUDA graph capture
• systemd.service(5) — specifically ExecStart argument quoting behaviour
• turboderp/Qwen3.5-9B-exl3 — ExLlamaV3 quants for the same model (alternative backend if vLLM proves to be a ceiling)

Ta ta for now, and I hope this saves someone the same afternoon of flag-tweaking.

Speed vs Quality: Comparing LFM2, Qwen3, and DeepSeek-R1 on a Local APU

In a previous post qwen3:30b-a3b was benchmarked at 12 TPS on a AMD APU (Ryzen 5800U, 64GB DDR4).  This beat every other model for token throughput on the system by exploiting Mixture-of-Experts to keep active parameters low.  Then lfm2:24b-a2b arrived and hit 17.8 TPS — 47% faster — using a hybrid SSM+MoE architecture.

But remember 'It depends!': tokens per second is just one metric.  A model that answers faster but worse is not a better model.  This post compares three architecturally distinct models on the same hardware across reasoning, synthesis, and analytical tasks.

---

The Models

Model	Architecture	TPS (Matt-Mini)	What makes it distinctive
lfm2:24b-a2b	Hybrid SSM+MoE	17.8	No KV cache (SSM), ~2B active params
qwen3:30b-a3b	MoE Transformer	12.0	3B active params, built-in thinking mode
deepseek-r1:14b	Dense Transformer	3.1	Reinforcement-learning trained reasoner

All running on Matt-Mini: AMD Ryzen 5800U, 64GB DDR4 (~50 GB/s bandwidth), AMD Vega 8 iGPU via Ollama's Vulkan backend.

---

Test 1: Formal Logic (Einstein's Puzzle)

The classic five-houses logic puzzle — 15 clues, five attributes per house, one correct solution. This tests structured multi-step deduction: can the model hold state, backtrack when it hits a contradiction, and reach a definite answer?

LFM2:24b-a2b — 17.8 TPS

LFM2 engaged immediately with a clear systematic approach, correctly anchoring on the fixed clues (Norwegian in house 1, house 3 drinks milk, house 2 is blue), then working through colour placement. Crucially, when it hit a contradiction — incorrectly placing green at house 3 which conflicted with the coffee/milk clue — it identified the error and self-corrected:

"Wait — green house is house 3, drinks milk — but clue 5 says green house drinks coffee. Contradiction! So our earlier assumption must be wrong. Let's recheck green/white placement."

It then correctly revised to green=4, white=5, and continued building the solution. At 2000 tokens it was still mid-deduction (the puzzle typically requires 2500–3500 tokens to complete), but the reasoning quality was coherent throughout with no hallucinated leaps. The self-correction under contradiction is the key capability here — many smaller models simply assert a wrong answer.

Qwen3:30b-a3b — 12.0 TPS

Qwen3's thinking mode is enabled by default. Given a 2000-token budget, the model consumed all 2000 tokens in internal reasoning and produced no visible output. This is not a failure of reasoning — it is a consequence of extended chain-of-thought: Qwen3 thinks before it writes, and for a complex puzzle, that thinking budget was exhausted before the answer appeared.

The practical implication: At 12 TPS, giving Qwen3 enough tokens to complete a hard logic problem (say 4000 tokens total — 2000 thinking, 2000 answer) means waiting ~5–6 minutes with nothing visible on screen until the model finishes thinking. For interactive use, this requires either disabling thinking mode (/no_think suffix or think: false in the API) or accepting the latency.

DeepSeek-R1:14b — 3.1 TPS

Same issue, worse TPS. At 3.1 TPS, 4000 tokens takes 21 minutes. DeepSeek-R1 is a dedicated reasoning model — the chain-of-thought is the point — but on bandwidth-constrained hardware the combination of dense architecture (no MoE savings), slow TPS, and long thinking chains makes it painful for interactive reasoning tasks. It is the right tool for problems where correctness matters more than speed and you are willing to wait.

---

The Thinking-Model Problem on Slow Hardware

This test uncovered a fundamental tension that does not appear in benchmark numbers:

Thinking models have a token overhead that multiplies with your hardware's latency.

On a GPU running at 50 TPS, Qwen3 spending 1000 tokens on internal reasoning costs 20 seconds. On an APU at 12 TPS, the same thinking costs 83 seconds — and you see nothing during that time. DeepSeek-R1 at 3.1 TPS spending 2000 tokens thinking costs 10.7 minutes of silence.

This creates a counterintuitive outcome: LFM2, which does not use extended chain-of-thought by default, feels smarter in interactive use on slow hardware — not because it reasons better, but because it shows its work in real-time rather than batching it invisibly. The perceived quality advantage is partly architectural and partly latency psychology.

For batch or automated tasks (where you submit a prompt and retrieve the answer later), thinking models regain their advantage. For interactive research assistance where you are iterating on prompts, fast transparent reasoning often beats slow invisible reasoning.

---

Test 2: Research Synthesis

Prompt: Advise on running the best LLM for general-purpose research on an AMD APU (Ryzen 5800U, 64GB DDR4, ~50 GB/s). Compare (a) large MoE with few active params, (b) hybrid SSM+MoE, (c) small dense transformer. Cover speed, quality, context handling, and give a recommendation.

Note: Qwen3 and DeepSeek-R1 were run with thinking disabled (think: false) so their responses are direct rather than chain-of-thought.

LFM2:24b-a2b — 17.8 TPS

LFM2 gave a well-structured response covering all three categories with accurate technical descriptions. On MoE it correctly identified the active-parameter advantage and low-bandwidth benefit. On SSM+MoE it correctly described the KV cache elimination benefit. On dense transformers it correctly explained the bandwidth bottleneck.

One notable weakness: it mentioned "AMD XDNA or ROCm kernels for optimal speed" — technically inaccurate for an Ollama/Vulkan setup. It conflated the architecture's theoretical requirements with platform-specific details it was uncertain about. This is a common failure mode in synthesis tasks: confident-sounding specifics that are slightly wrong.

Qwen3:30b-a3b — 12.0 TPS (thinking disabled)

With thinking disabled, Qwen3 produced a response that read as an externalised internal monologue — thinking-as-prose rather than a structured answer. It walked through the problem step-by-step in first person ("Let me break this down... Wait, the user said..."), ultimately recommending a small dense model like Mistral 7B.

This recommendation is wrong for this specific hardware, and interestingly, Qwen3 is itself running on this hardware. It gave generic advice calibrated to typical CPU inference bottlenecks rather than the actual measured results. It also suggested llama.cpp "15–25 TPS on a 5800U for a 7B Q4_K_M model" — a reasonable estimate for CPU-only inference, but ignoring the iGPU Vulkan backend that gives the measured 12 TPS for a 30B MoE model.

The knowledge cutoff problem: Qwen3 has no training data on LFM2 or the specific benchmark results for this hardware. It gave the best answer it could from general principles, and those principles pointed it toward the wrong conclusion. The model recommended a 7B dense model when the empirical data shows a 24B hybrid SSM+MoE outperforms it by a significant margin on this exact configuration.

DeepSeek-R1:14b — 3.1 TPS (thinking disabled)

DeepSeek-R1's response was clean and well-structured, but contained a clear hallucination: it described SSM as "Switched Sparse Memory" — a fabricated expansion of the acronym. SSM stands for State Space Model, a fundamentally different concept.

Its recommendation — large MoE transformer — was directionally correct, citing that "~50 GB/s is high bandwidth." This is precisely backwards: 50 GB/s is the bottleneck on this hardware, not an advantage. The model interpreted the bandwidth figure in absolute terms rather than relative to the model's memory requirements.

Despite these errors, the structural reasoning was sound: MoE's active-parameter reduction is the right lever to pull on bandwidth-constrained hardware. It got to the right answer via partially wrong reasoning.

---

Test 3: Critical Analysis

Prompt: Analyse the claim: "Q4 always reduces model quality compared to Q8, so for serious research you should always use Q8." Give a rigorous assessment.

This is a well-defined analytical task with a clear correct answer. All three models had relevant training data. This is where quality differences should be most visible.

LFM2:24b-a2b — 17.8 TPS

LFM2 identified three flaws in the claim:

1. Universal claim without qualification — Q4 vs Q8 impact depends on architecture, task, calibration, and hardware. "Always" is unjustified.
2. Ignores hardware acceleration — AVX_VNNI provides INT8 dot product acceleration; on AVX_VNNI CPUs, Q8_0 carries no compute overhead, changing the tradeoff entirely.
3. Task-specific sensitivity — Coarse reasoning is less sensitive to precision loss than fine-grained factual recall.

The hardware-specific point about AVX_VNNI is precisely the kind of nuance that matters for the actual decision. This was accurate and practically useful.

Qwen3:30b-a3b — 12.0 TPS (thinking disabled)

Qwen3 gave the most thorough response of the three, leading with the logical structure: the word "always" is a universal quantifier that is falsified by a single counterexample. It cited concrete benchmark evidence — Q4 on Llama 3 and Mistral models incurs less than 1% absolute accuracy loss on MMLU vs Q8 — and identified multiple factors the claim ignores: quantisation-aware training, task sensitivity, model architecture, and implicit regularisation effects.

It also identified a counterintuitive scenario: some models fine-tuned with quantisation-aware training show no significant degradation at Q4, or in edge cases slight improvements due to regularisation. This is a more complete analysis than LFM2's.

The response was cut at 1000 tokens mid-sentence, suggesting there was more to come. The quality of what was produced was high.

DeepSeek-R1:14b — 3.1 TPS (thinking disabled)

DeepSeek-R1 produced the most concise response. It correctly identified the binary framing as the flaw (Q4 offers 16 values, Q8 offers 256 — but this alone doesn't determine quality impact), and noted that the practical effect depends on the model and task. The response was shorter and less detailed than the others, but accurate within its scope.

At 3.1 TPS, the time cost of a thorough analysis at DeepSeek-R1's depth is high. For tasks where analysis quality scales with thoroughness, the slow TPS compounds against you.

---

Conclusion

Factual Accuracy

All three models made errors on the synthesis task — LFM2 got a platform detail wrong, Qwen3 gave the wrong recommendation, DeepSeek-R1 hallucinated an acronym expansion. The analysis task showed higher accuracy across all three, because the claim being analysed is well within their training distribution. Models are more reliable on tasks that resemble their training data. Novel hardware configurations and cutting-edge model architectures fall outside that zone.

Reasoning Quality

For the logic puzzle, LFM2's transparent step-by-step reasoning with explicit self-correction was more useful interactively than Qwen3's silent exhausted thinking budget. On the analysis task, Qwen3 produced the most thorough and structured response when thinking was disabled — the base model quality is high when it actually produces output.

Thinking-mode Tradeoff

Thinking mode is a quality multiplier. But on hardware where generation is slow, it is also a latency multiplier — and one that applies silently before you see any output. The practical rule for APU-class hardware:

• Interactive use: Disable thinking (think: false or /no_think). You get the answer faster, and for most tasks the quality loss is modest.
• Batch / overnight analysis: Enable thinking and set a high token budget. You submit the job, come back later, get a more thorough answer.

LFM2 sidesteps this entirely: its architecture doesn't separate thinking from output, so you see the reasoning in real-time as it generates.

---

Summary: When to Use Each Model

Scenario	Best choice	Why
Interactive research, iterative queries	lfm2:24b-a2b	Fastest, transparent reasoning, good synthesis
Hard reasoning, batch mode	deepseek-r1:14b	Dedicated reasoner; worth the wait
Thorough structured analysis, batch mode	qwen3:30b-a3b (thinking enabled)	Best analysis quality when given token budget
Code generation	qwen3-coder:30b-a3b-q4_K_M	Specialised fine-tune, 12 TPS
Long document analysis	lfm2:24b-a2b	SSM avoids KV cache growth at long context
Uncensored / sensitive research	qwen3.5-abliterated:35b-a3b-q4_K	No guardrails, 4.65 TPS
Quick simple queries	qwen3:8b	5.3 TPS, low overhead

Speed and quality are not independent on bandwidth-constrained hardware. A model that is four times slower doesn't just cost you time — it changes the nature of the interaction.  Thinking modes that are near-free on a GPU become minutes-long commitments on an APU, turning iterative exploration into batch processing. LFM2's hybrid SSM+MoE architecture produces the best combination of speed and quality for interactive use on this hardware, not because it is the most capable model in isolation, but because it delivers its capability at a speed that keeps research workflows fluid.

---

Appendix: Hardware and Setup

Matt-Mini:
- CPU: AMD Ryzen 7 5800U (Zen 3, no AVX_VNNI)
- iGPU: AMD Radeon Vega 8 (shared DDR4, ~50 GB/s)
- RAM: 64GB DDR4-3200
- Inference: Ollama 0.20.6 + Vulkan backend

API note: Thinking can be disabled per-request via "think": false in the Ollama generate payload, or by appending /no_think to the prompt for Qwen3 models. DeepSeek-R1 respects think: false at the API level.

Tested April 2026.

Speed vs Quality: Comparing LFM2, Qwen3, and DeepSeek-R1 on a Local APU

The Question

In a previous post I found that qwen3:30b-a3b runs at 12 TPS on my AMD APU (Ryzen 5800U, 64GB DDR4), beating every other model on the system by exploiting Mixture-of-Experts to keep active parameters low. Then lfm2:24b-a2b arrived and hit 17.8 TPS — 47% faster — using a hybrid SSM+MoE architecture.

But tokens per second is only half the story. A model that answers faster but worse is not a better model. This post compares three architecturally distinct models on the same hardware across reasoning, synthesis, and analytical tasks.

---

The Contenders

Model	Architecture	TPS (Matt-Mini)	What makes it distinctive
lfm2:24b-a2b	Hybrid SSM+MoE	17.8	No KV cache (SSM), ~2B active params
qwen3:30b-a3b	MoE Transformer	12.0	3B active params, built-in thinking mode
deepseek-r1:14b	Dense Transformer	3.1	Reinforcement-learning trained reasoner

All running on Matt-Mini: AMD Ryzen 5800U, 64GB DDR4 (~50 GB/s bandwidth), AMD Vega 8 iGPU via Ollama's Vulkan backend.

---

Test 1: Formal Logic (Einstein's Puzzle)

The classic five-houses logic puzzle — 15 clues, five attributes per house, one correct solution. This tests structured multi-step deduction: can the model hold state, backtrack when it hits a contradiction, and reach a definite answer?

LFM2:24b-a2b — 17.8 TPS

LFM2 engaged immediately with a clear systematic approach, correctly anchoring on the fixed clues (Norwegian in house 1, house 3 drinks milk, house 2 is blue), then working through colour placement. Crucially, when it hit a contradiction — incorrectly placing green at house 3 which conflicted with the coffee/milk clue — it identified the error and self-corrected:

"Wait — green house is house 3, drinks milk — but clue 5 says green house drinks coffee. Contradiction! So our earlier assumption must be wrong. Let's recheck green/white placement."

It then correctly revised to green=4, white=5, and continued building the solution. At 2000 tokens it was still mid-deduction (the puzzle typically requires 2500–3500 tokens to complete), but the reasoning quality was coherent throughout with no hallucinated leaps. The self-correction under contradiction is the key capability here — many smaller models simply assert a wrong answer.

Qwen3:30b-a3b — 12.0 TPS

Qwen3's thinking mode is enabled by default. Given a 2000-token budget, the model consumed all 2000 tokens in internal reasoning and produced no visible output. This is not a failure of reasoning — it is a consequence of extended chain-of-thought: Qwen3 thinks before it writes, and for a complex puzzle, that thinking budget was exhausted before the answer appeared.

The practical implication: At 12 TPS, giving Qwen3 enough tokens to complete a hard logic problem (say 4000 tokens total — 2000 thinking, 2000 answer) means waiting ~5–6 minutes with nothing visible on screen until the model finishes thinking. For interactive use, this requires either disabling thinking mode (/no_think suffix or think: false in the API) or accepting the latency.

DeepSeek-R1:14b — 3.1 TPS

Same issue, worse TPS. At 3.1 TPS, 4000 tokens takes 21 minutes. DeepSeek-R1 is a dedicated reasoning model — the chain-of-thought is the point — but on bandwidth-constrained hardware the combination of dense architecture (no MoE savings), slow TPS, and long thinking chains makes it painful for interactive reasoning tasks. It is the right tool for problems where correctness matters more than speed and you are willing to wait.

---

The Thinking-Model Problem on Slow Hardware

This test uncovered a fundamental tension that does not appear in benchmark numbers:

Thinking models have a token overhead that multiplies with your hardware's latency.

On a GPU running at 50 TPS, Qwen3 spending 1000 tokens on internal reasoning costs 20 seconds. On an APU at 12 TPS, the same thinking costs 83 seconds — and you see nothing during that time. DeepSeek-R1 at 3.1 TPS spending 2000 tokens thinking costs 10.7 minutes of silence.

This creates a counterintuitive outcome: LFM2, which does not use extended chain-of-thought by default, feels smarter in interactive use on slow hardware — not because it reasons better, but because it shows its work in real-time rather than batching it invisibly. The perceived quality advantage is partly architectural and partly latency psychology.

For batch or automated tasks (where you submit a prompt and retrieve the answer later), thinking models regain their advantage. For interactive research assistance where you are iterating on prompts, fast transparent reasoning often beats slow invisible reasoning.

---

Test 2: Research Synthesis

Prompt: Advise on running the best LLM for general-purpose research on an AMD APU (Ryzen 5800U, 64GB DDR4, ~50 GB/s). Compare (a) large MoE with few active params, (b) hybrid SSM+MoE, (c) small dense transformer. Cover speed, quality, context handling, and give a recommendation.

Note: Qwen3 and DeepSeek-R1 were run with thinking disabled (think: false) so their responses are direct rather than chain-of-thought.

LFM2:24b-a2b — 17.8 TPS

LFM2 gave a well-structured response covering all three categories with accurate technical descriptions. On MoE it correctly identified the active-parameter advantage and low-bandwidth benefit. On SSM+MoE it correctly described the KV cache elimination benefit. On dense transformers it correctly explained the bandwidth bottleneck.

One notable weakness: it mentioned "AMD XDNA or ROCm kernels for optimal speed" — technically inaccurate for an Ollama/Vulkan setup. It conflated the architecture's theoretical requirements with platform-specific details it was uncertain about. This is a common failure mode in synthesis tasks: confident-sounding specifics that are slightly wrong.

Qwen3:30b-a3b — 12.0 TPS (thinking disabled)

With thinking disabled, Qwen3 produced a response that read as an externalised internal monologue — thinking-as-prose rather than a structured answer. It walked through the problem step-by-step in first person ("Let me break this down... Wait, the user said..."), ultimately recommending a small dense model like Mistral 7B.

This recommendation is wrong for this specific hardware, and interestingly, Qwen3 is itself running on this hardware. It gave generic advice calibrated to typical CPU inference bottlenecks rather than the actual measured results. It also suggested llama.cpp "15–25 TPS on a 5800U for a 7B Q4_K_M model" — a reasonable estimate for CPU-only inference, but ignoring the iGPU Vulkan backend that gives the measured 12 TPS for a 30B MoE model.

The knowledge cutoff problem: Qwen3 has no training data on LFM2 or the specific benchmark results for this hardware. It gave the best answer it could from general principles, and those principles pointed it toward the wrong conclusion. The model recommended a 7B dense model when the empirical data shows a 24B hybrid SSM+MoE outperforms it by a significant margin on this exact configuration.

DeepSeek-R1:14b — 3.1 TPS (thinking disabled)

DeepSeek-R1's response was clean and well-structured, but contained a clear hallucination: it described SSM as "Switched Sparse Memory" — a fabricated expansion of the acronym. SSM stands for State Space Model, a fundamentally different concept.

Its recommendation — large MoE transformer — was directionally correct, citing that "~50 GB/s is high bandwidth." This is precisely backwards: 50 GB/s is the bottleneck on this hardware, not an advantage. The model interpreted the bandwidth figure in absolute terms rather than relative to the model's memory requirements.

Despite these errors, the structural reasoning was sound: MoE's active-parameter reduction is the right lever to pull on bandwidth-constrained hardware. It got to the right answer via partially wrong reasoning.

---

Test 3: Critical Analysis

Prompt: Analyse the claim: "Q4 always reduces model quality compared to Q8, so for serious research you should always use Q8." Give a rigorous assessment.

This is a well-defined analytical task with a clear correct answer. All three models had relevant training data. This is where quality differences should be most visible.

LFM2:24b-a2b — 17.8 TPS

LFM2 identified three flaws in the claim:

1. Universal claim without qualification — Q4 vs Q8 impact depends on architecture, task, calibration, and hardware. "Always" is unjustified.
2. Ignores hardware acceleration — AVX_VNNI provides INT8 dot product acceleration; on AVX_VNNI CPUs, Q8_0 carries no compute overhead, changing the tradeoff entirely.
3. Task-specific sensitivity — Coarse reasoning is less sensitive to precision loss than fine-grained factual recall.

The hardware-specific point about AVX_VNNI is precisely the kind of nuance that matters for the actual decision. This was accurate and practically useful.

Qwen3:30b-a3b — 12.0 TPS (thinking disabled)

Qwen3 gave the most thorough response of the three, leading with the logical structure: the word "always" is a universal quantifier that is falsified by a single counterexample. It cited concrete benchmark evidence — Q4 on Llama 3 and Mistral models incurs less than 1% absolute accuracy loss on MMLU vs Q8 — and identified multiple factors the claim ignores: quantisation-aware training, task sensitivity, model architecture, and implicit regularisation effects.

It also identified a counterintuitive scenario: some models fine-tuned with quantisation-aware training show no significant degradation at Q4, or in edge cases slight improvements due to regularisation. This is a more complete analysis than LFM2's.

The response was cut at 1000 tokens mid-sentence, suggesting there was more to come. The quality of what was produced was high.

DeepSeek-R1:14b — 3.1 TPS (thinking disabled)

DeepSeek-R1 produced the most concise response. It correctly identified the binary framing as the flaw (Q4 offers 16 values, Q8 offers 256 — but this alone doesn't determine quality impact), and noted that the practical effect depends on the model and task. The response was shorter and less detailed than the others, but accurate within its scope.

At 3.1 TPS, the time cost of a thorough analysis at DeepSeek-R1's depth is high. For tasks where analysis quality scales with thoroughness, the slow TPS compounds against you.

---

What the Tests Reveal

On factual accuracy

All three models made errors on the synthesis task — LFM2 got a platform detail wrong, Qwen3 gave the wrong recommendation, DeepSeek-R1 hallucinated an acronym expansion. The analysis task showed higher accuracy across all three, because the claim being analysed is well within their training distribution. Models are more reliable on tasks that resemble their training data. Novel hardware configurations and cutting-edge model architectures fall outside that zone.

On reasoning quality

For the logic puzzle, LFM2's transparent step-by-step reasoning with explicit self-correction was more useful interactively than Qwen3's silent exhausted thinking budget. On the analysis task, Qwen3 produced the most thorough and structured response when thinking was disabled — the base model quality is high when it actually produces output.

On the thinking-mode tradeoff

Thinking mode is a quality multiplier. But on hardware where generation is slow, it is also a latency multiplier — and one that applies silently before you see any output. The practical rule for APU-class hardware:

• Interactive use: Disable thinking (think: false or /no_think). You get the answer faster, and for most tasks the quality loss is modest.
• Batch / overnight analysis: Enable thinking and set a high token budget. You submit the job, come back later, get a more thorough answer.

LFM2 sidesteps this entirely: its architecture doesn't separate thinking from output, so you see the reasoning in real-time as it generates.

---

Summary: When to Use Each Model

Scenario	Best choice	Why
Interactive research, iterative queries	lfm2:24b-a2b	Fastest, transparent reasoning, good synthesis
Hard reasoning, batch mode	deepseek-r1:14b	Dedicated reasoner; worth the wait
Thorough structured analysis, batch mode	qwen3:30b-a3b (thinking enabled)	Best analysis quality when given token budget
Code generation	qwen3-coder:30b-a3b-q4_K_M	Specialised fine-tune, 12 TPS
Long document analysis	lfm2:24b-a2b	SSM avoids KV cache growth at long context
Uncensored / sensitive research	qwen3.5-abliterated:35b-a3b-q4_K	No guardrails, 4.65 TPS
Quick simple queries	qwen3:8b	5.3 TPS, low overhead

The key finding

Speed and quality are not independent on bandwidth-constrained hardware. A model that is four times slower doesn't just cost you time — it changes the nature of the interaction. Thinking modes that are near-free on a GPU become minutes-long commitments on an APU, turning iterative exploration into batch processing. LFM2's hybrid SSM+MoE architecture produces the best combination of speed and quality for interactive use on this hardware, not because it is the most capable model in isolation, but because it delivers its capability at a speed that keeps research workflows fluid.

---

Appendix: Hardware and Setup

Matt-Mini:
- CPU: AMD Ryzen 7 5800U (Zen 3, no AVX_VNNI)
- iGPU: AMD Radeon Vega 8 (shared DDR4, ~50 GB/s)
- RAM: 64GB DDR4-3200
- Inference: Ollama 0.20.6 + Vulkan backend

API note: Thinking can be disabled per-request via "think": false in the Ollama generate payload, or by appending /no_think to the prompt for Qwen3 models. DeepSeek-R1 respects think: false at the API level.

Tested April 2026.