I Wasted a Week Running Ollama on K3s Before Understanding GPU Pass-Through

I remember the exact moment I realized my "private AI infrastructure" was a lie. I'd spun up K3s on a server with an RTX 4090, deployed Ollama with Open-WebUI, and asked it a simple question. Then I watched the cursor blink. One token per second. The GPU sat there, idle, while my CPU melted into the motherboard trying to run a 13B model on pure arithmetic.

I'd made a rookie mistake: I assumed containerization was transparent. That Kubernetes would just... work. I spent an entire week tuning memory limits, adjusting worker threads, and blaming Ollama's optimization before realizing the brutal truth, my pod couldn't even see the GPU.

This is a problem I see repeated constantly in the community. People build impressive local LLM stacks, deploy them to Kubernetes, and watch everything fall apart. The fix exists, but it requires understanding a stack most developers don't think about: how hardware isolation actually works in containers.

The Isolation Problem Nobody Talks About

Here's what actually happens when you run Ollama in K3s without GPU configuration: your container is intentionally blind to the host's PCI hardware. That RTX 4090? It doesn't exist from inside the pod's perspective. The NVIDIA device nodes (/dev/nvidia0, /dev/nvidia-uvm) are invisible.

Ollama, being pragmatic, detects no CUDA runtime and falls back to CPU inference. This isn't a failure, it's a safety feature. But it's also a catastrophe for performance. CPU matrix multiplication for a 13B model means constant, high-latency memory transfers between system RAM and CPU cache. You're essentially asking your processor to do the GPU's entire job with one hand tied behind its back.

The issue isn't K3s or Ollama specifically. It's that most Kubernetes deployments never needed GPU access before. The ecosystem built itself around CPU workloads. GPU support feels like an afterthought because, honestly, it kind of is.

Building the Hardware Bridge: Three Layers

To fix this, you need three specific components working together. I think of it as a bridge from the Linux kernel driver all the way into your pod's namespace.

Layer 1 is your NVIDIA drivers and the Container Toolkit on the host. This is the foundation. Without proper drivers and the Container Toolkit installed, nothing else matters. The Container Toolkit teaches containerd how to inject GPU libraries into container specs.

Layer 2 is the NVIDIA Kubernetes Device Plugin. This plugin runs as a daemonset on your cluster and tells the Kubernetes scheduler that GPUs are available. It's how pods can actually request GPU resources in their manifest.

Layer 3 is your pod configuration. You specify nvidia.com/gpu: "1" in your resource limits, and suddenly the scheduler knows to place your pod on a node with available GPUs and wire everything up.

The magic happens at the intersection of these three layers. When done correctly, Ollama inside the pod gets full access to CUDA, VRAM, and can run at 60-80 tokens/second instead of 1.

What I'd Do Differently

The original article is technically sound, but I want to be honest about one thing: this setup still feels fragile to me. The K3s containerd configuration requires template files that survive service restarts. You can't just edit configs directly, K3s overwrites them. I've had this bite me before.

My approach now is more conservative. I verify each layer independently. First, I confirm nvidia-smi works on the host. Then I test GPU access with a simple container (nvidia/cuda:latest) before touching Ollama. Only then do I deploy the Device Plugin and my actual workloads.

I also stay paranoid about driver versions. NVIDIA's compatibility matrix is unforgiving. A driver upgrade that breaks CUDA compatibility will silently fall back to CPU mode, and you might not notice for hours.

The other thing I'd emphasize: this is genuinely necessary for any local LLM work. CPU inference isn't just slow, it's economically irrational. You're paying for hardware you can't use.

Practical Setup Check

Here's the minimal verification flow I use:

# 1. Host GPU check
nvidia-smi

# 2. Container GPU check
docker run --rm --gpus all nvidia/cuda:12.0.1-base-ubuntu22.04 nvidia-smi

# 3. K3s Device Plugin check
kubectl get nodes -o json | jq '.items[].status.capacity'
# Should show "nvidia.com/gpu": "1" (or however many GPUs you have)

# 4. Pod GPU allocation
kubectl describe pod <ollama-pod> | grep -A 5 "Limits"
# Should show nvidia.com/gpu: 1

If any of these fail, you haven't successfully built the bridge yet.

Next Steps

If you're running Ollama on K3s and seeing single-digit token speeds, stop assuming it's a software problem. It's almost certainly a hardware visibility problem. Work backwards from that nvidia-smi check I mentioned. Verify each layer exists before moving to the next.

What's your setup like? Are you running into this problem right now, or have you already worked through it? I'd genuinely like to know what issues came up for you during the process.

Source: This post was inspired by "Why Your Ollama LLMs Are Running Painfully Slow on K3s" by Dev.to. Read the original article