Running multiple GPU-accelerated services on a single consumer GPU is a juggling act. This post documents how I diagnosed a VRAM exhaustion problem with four concurrent Kokoro TTS instances, why the "official" fix didn't exist, and how I patched the server to load the model on demand and release VRAM after 120 seconds of inactivity.
The Setup
I'm running four variants of kokoro-fastapi in Incus LXC containers, each serving a different combination of device and streaming mode:
| Hostname | Container | Mode |
|---|---|---|
kokoro-gpu-streaming.lan |
kokoro-stream-gpu |
GPU + streaming patches |
kokoro-gpu.lan |
kokoro-gpu |
GPU, unpatched |
kokoro-cpu-streaming.lan |
kokoro-stream |
CPU + streaming patches |
kokoro-cpu.lan |
kokoro-tts |
CPU, unpatched |
Each one is reverse-proxied by Caddy, so all four are reachable by name with no port numbers.
The Problem: 8 GB Doesn't Go Far
After getting the GPU streaming instance running cleanly, I tried to start the GPU non-streaming instance. It crash-looped immediately.
Apr 17 07:55:11 kokoro-gpu uv[402]: RuntimeError: Warmup failed: Failed to load model:
Failed to load Kokoro model: CUDA out of memory. Tried to allocate 2.00 MiB.
GPU 0 has a total capacity of 7.62 GiB of which 3.75 MiB is free.
Process 2729035 has 174.00 MiB memory in use. ← ComfyUI
Process 2750368 has 880.00 MiB memory in use. ← kokoro-stream-gpu
Process 2763628 has 6.26 GiB memory in use. ← Ollama (model loaded)
Process 2766828 has 296.00 MiB memory in use.
Running nvidia-smi made the picture clear:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09 Driver Version: 580.126.09 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| 0 NVIDIA GeForce RTX 4070 ... Off | 00000000:01:00.0 Off| N/A |
| N/A 56C P4 9W / 40W | 7501MiB / 8188MiB | 0% Default |
+-----------------------------------------+------------------------+----------------------+
| 0 N/A N/A 2729035 C ...ComfyUI/.venv/bin/python3 174MiB |
| 0 N/A N/A 2750368 C /app/.venv/bin/python3 880MiB |
| 0 N/A N/A 2763628 C /usr/local/bin/ollama 6414MiB |
7,501 MiB of 7,623 MiB used. Ollama had a model loaded and was sitting idle, consuming 6.4 GB with nothing to do.
Each Kokoro GPU instance needs ~880 MB. Two instances plus ComfyUI plus an idle Ollama model adds up to more than 8 GB. Something had to give.
Mitigation Options
Before writing any code, it's worth considering what levers exist:
1. Reduce the number of GPU processes. Stop ComfyUI or unload the Ollama model when not in use. Ollama exposes a keep_alive API parameter — setting it to 0 on any request tells Ollama to unload immediately after responding. This frees 6.4 GB instantly, enough for both Kokoro instances with room to spare. It just means Ollama has a cold-start penalty on its next request too.
2. Use CPU for some instances. Kokoro on CPU takes ~1–3 seconds for a short phrase instead of ~90ms on GPU. For benchmarking purposes — which is what this setup is for — that's acceptable. The non-streaming instance doesn't need GPU to produce valid benchmark data.
3. Lazy-load the model. The server starts and listens, but only loads weights into VRAM when the first request arrives. Combined with an idle timeout that unloads the model after a period of inactivity, multiple GPU services can share the same card as long as they're not all active simultaneously.
For this setup the goal was four genuinely independent instances, so lazy loading was the right answer. Before implementing it, though, there was a detour.
3a. Checking What AI Agents Tell You
This setup is managed by AI agents (Claude), and the agents suggested a quick fix before any code was read:
Set
IDLE_TIMEOUT=120andWARMUP_STEPS=0in the systemd service unit.WARMUP_STEPS=0prevents the model loading at startup, andIDLE_TIMEOUTunloads it after inactivity.
This sounds authoritative. It's also completely wrong — and importantly, it isn't a hallucination in the technical sense. The model didn't confabulate random tokens. It generated a plausible-sounding answer based on patterns from similar projects (many ML serving frameworks do have env vars like these), then stated it as fact without checking whether this particular codebase actually implemented them.
The distinction matters: a hallucination is making something up. This was something subtly different — confident assertion without verification. The env vars were set:
Environment=IDLE_TIMEOUT=120
Environment=WARMUP_STEPS=0
The service still OOM'd on startup. The next step was what should have happened first:
grep -r "IDLE_TIMEOUT\|WARMUP" /app/api/
# (no output)
Not a single match. These variables are read nowhere in the codebase. The model always loads at startup, unconditionally, regardless of any environment variable you set. Version 0.3.0 simply doesn't have this feature.
The lesson isn't that AI agents are unreliable — they're genuinely useful for this kind of infrastructure work. The lesson is that suggestions about third-party library behaviour need to be verified against the actual source before being acted on, especially when they're offered without a citation. "Check the docs / grep the source" takes thirty seconds and would have saved the detour entirely. A good agent should do this itself before making the suggestion; when it doesn't, the human needs to catch it.
Diagnosing the Startup Flow
The relevant code lives in two files. First, main.py starts everything in a FastAPI lifespan context:
# api/src/main.py — original startup (simplified)
@asynccontextmanager
async def lifespan(app: FastAPI):
model_manager = await get_manager()
voice_manager = await get_voice_manager()
# This loads the model weights immediately at startup
device, model, voicepack_count = await model_manager.initialize_with_warmup(
voice_manager
)
yield
initialize_with_warmup in model_manager.py calls initialize() → load_model() → runs a warmup inference pass. The model is in VRAM before the first request ever arrives.
The generate() method had no concept of loading on demand:
# original generate() — raises if backend not initialized
async def generate(self, *args, **kwargs):
if not self._backend:
raise RuntimeError("Backend not initialized")
async for chunk in self._backend.generate(*args, **kwargs):
yield chunk
The Fix: Lazy Loading + Idle Unload
The fix has two parts: skip model loading at startup, and load it on the first generate() call. A background coroutine checks every 10 seconds and unloads if the model has been idle for 120 seconds.
Part 1 — main.py: Skip startup model loading
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Lifespan context manager — model loads on first request, not here."""
import asyncio as _asyncio
from .inference.model_manager import get_manager
from .inference.voice_manager import get_manager as get_voice_manager
from .services.temp_manager import cleanup_temp_files
await cleanup_temp_files()
# Set up the manager objects but do NOT load model weights.
# The model will load on the first call to generate().
model_manager = await get_manager()
voice_manager = await get_voice_manager()
# Start the background coroutine that watches for idle time
# and unloads the model from VRAM when it hasn't been used.
idle_task = _asyncio.create_task(model_manager.idle_monitor())
logger.info("Kokoro TTS ready — model will load on first request")
yield # Server runs here
idle_task.cancel() # Clean up on shutdown
The key change: initialize_with_warmup() is never called. The server starts, listens for requests, and uses zero VRAM.
Part 2 — model_manager.py: Lazy load + idle monitor
class ModelManager:
def __init__(self, config=None):
self._config = config or model_config
self._backend: Optional[KokoroV1] = None
self._device: Optional[str] = None
# Track when the model was last used (monotonic clock, not wall time)
self._last_used: Optional[float] = None
# Prevent two concurrent requests from both trying to load the model
self._loading: bool = False
# Unload after this many seconds of inactivity
self._idle_timeout: int = 120
async def _load_if_needed(self) -> None:
"""Load model weights into VRAM on demand.
Thread-safety note: if two requests arrive simultaneously before the
model is loaded, the _loading flag serialises them so only one
actually loads the model. The second waits, then sees the model is
already there and returns immediately.
"""
# Fast path: model already loaded
if (self._backend is not None
and hasattr(self._backend, '_model')
and self._backend._model is not None):
return
# Wait if another coroutine is already loading
while self._loading:
await asyncio.sleep(0.1)
# Re-check after waiting — the other coroutine may have loaded it
if (self._backend is not None
and hasattr(self._backend, '_model')
and self._backend._model is not None):
return
# This coroutine wins the race — load the model
self._loading = True
try:
logger.info("Loading model on demand...")
await self.initialize() # create KokoroV1 object
model_path = self._config.pytorch_kokoro_v1_file
await self.load_model(model_path) # load weights into VRAM
logger.info("Model loaded on demand")
finally:
self._loading = False # always release the lock
async def idle_monitor(self) -> None:
"""Background task: unload model after idle_timeout seconds of inactivity.
Checks every 10 seconds. If the model is loaded and hasn't been used
for 120 seconds, calls unload_all() to free VRAM. The model will
reload transparently on the next request.
"""
while True:
await asyncio.sleep(10) # poll interval
if self._backend is not None and self._last_used is not None:
idle = time.monotonic() - self._last_used
if idle >= self._idle_timeout:
logger.info(f"Idle for {idle:.0f}s — unloading model from VRAM")
self.unload_all() # calls backend.unload() and sets _backend = None
self._last_used = None # reset so we don't trigger again immediately
async def generate(self, *args, **kwargs):
"""Generate audio, loading the model first if needed."""
# Ensure model is in VRAM before we try to run inference
await self._load_if_needed()
# Update last-used timestamp so the idle monitor knows the model is active
self._last_used = time.monotonic()
try:
async for chunk in self._backend.generate(*args, **kwargs):
if settings.default_volume_multiplier != 1.0:
chunk.audio *= settings.default_volume_multiplier
yield chunk
# Keep updating during long generations (multi-sentence text)
self._last_used = time.monotonic()
except Exception as e:
raise RuntimeError(f"Generation failed: {e}")
Verification
After patching, the service starts with zero VRAM usage:
Apr 17 08:05:05 kokoro-stream-gpu uv[1767]: Kokoro TTS ready — model will load on first request
Apr 17 08:05:05 kokoro-stream-gpu uv[1767]: INFO: Application startup complete.
Confirming via nvidia-smi that the new process (host PID 2774231) doesn't appear in the GPU process list at all — zero VRAM allocated. The previous 812 MiB entry was a lingering process from the crash-loop cycle, cleaning itself up.
On the first request, the model loads in ~5 seconds and synthesis proceeds normally. After 120 seconds of inactivity, the idle monitor fires:
Idle for 121s — unloading model from VRAM
And the VRAM is returned to the system.
What This Enables
Both GPU instances now coexist. When neither has been called recently, they hold zero VRAM — Ollama and ComfyUI can use the full 8 GB uncontested. When a request comes in, whichever instance is needed loads its model (~5s cold start), handles the request, then unloads 120 seconds later.
If both GPU instances happen to be active simultaneously — unlikely in practice — they would compete for the ~1.1 GB of VRAM not occupied by ComfyUI, and one would OOM. That's an acceptable trade-off for a benchmarking setup. For a production single-instance deployment, the same patch gives you a TTS server that is invisible to the GPU scheduler when not in use.
Summary
- kokoro-fastapi 0.3.0 has no built-in lazy loading or idle unload — both are custom patches
- The startup model load is in
lifespan()inmain.py; removing it is one line - The lazy load logic belongs in
generate()with a flag to serialise concurrent first-requests - The idle monitor is a simple asyncio background task that polls every 10 seconds
- Both patches are forward-compatible:
initialize_with_warmup()is preserved in the codebase for anyone who needs eager loading, and the new_load_if_needed()/idle_monitor()sit alongside it
