The Quantization Problem I Didn't Know I Had Until I Tried Running LLMs Locally

I spent three weeks last month trying to serve a 26B parameter model on a single GPU in my test environment. I kept hitting the same wall: the memory just wouldn't cooperate. I'd quantize with one method, get mediocre results. Try another, burn tokens on latency. I was ready to give up and rent cloud inference like everyone else. Then I read about what Google did with Gemma 4's quantization-aware training and realized I'd been thinking about this problem completely backwards.

The issue isn't just that models are getting bigger. It's that we've been treating quantization like an afterthought, a compression hack applied after training finishes. But Google trained Gemma 4 with quantization baked into the entire process, which means the weights already know they're supposed to live in 4-bit space. That's fundamentally different, and the performance gap proves it.

The Real Problem: Not All Quantization Is Created Equal

Here's what most people don't understand: when you take a model trained in full precision and quantize it afterward, you're asking weights that were trained to live in a continuous space to suddenly compress into discrete levels. It works, but you're leaving performance on the table.

Quantization-aware training (QAT) does this differently. During training, Google simulated the quantization process itself, so the model learned to express itself accurately within 4-bit constraints from the start. When the training finished, the weights were already living on a 16-level grid per group of 32 values.

The article's author (xbill9) discovered something clever here: Google exported these QAT weights in bf16 format, which sounds redundant until you realize the values themselves are already quantized, just stored in a higher precision container. The actual work was recovering the correct quantization scale for each group, not re-quantizing from scratch.

Building the Missing Piece

The core contribution here is solving a specific gap in vLLM's TPU support. Google published QAT checkpoints for most Gemma 4 sizes, but the 26B model had no optimized vLLM build for single-chip inference. RedHat's FP8 version existed, but it only held 3,456 tokens of KV cache. The author reverse-engineered the quantization metadata from Google's export and repacked it into vLLM's native Compressed Tensors format.

This isn't trivial work. The repack script has to:

1. Read the "unquantized" bf16 checkpoint
2. Recover the original quantization grid for each group (trying m values from 1-8 to find which one reproduces the exact weights)
3. Refine via least squares fitting
4. Rewrite into a format vLLM's JAX backend can actually use

The verification step is where I got impressed: byte-checking against the original to catch any drift, then comparing quantization levels to confirm the training signal stayed intact.

What This Means for My Production Work

I'm going to be honest: I've been lazy about quantization. I reach for 8-bit when I need to fit something on one GPU, accept the latency hit, and move on. But this article showed me that with QAT models, I don't have to choose between serving density and latency anymore.

The numbers are striking. The QAT Gemma 4 26B fit in 17.43 GiB on a TPU v6e (versus 27.99 for FP8), with nearly 16x more KV cache capacity and almost 2x the throughput. On a classification benchmark, the accuracy drop was minimal (0.4-1.1 points) compared to full precision. That's the kind of trade I'd take every time.

What concerns me is the implementation overhead. This required deep knowledge of vLLM's TPU backend, custom JAX code, and essentially reconstructing Google's internal quantization metadata from exported weights. Most teams can't do this. We need these builds to come pre-baked from the model authors or to be handled transparently by inference frameworks.

My Take: The Infrastructure Gap Is Real

The real story here isn't just "quantization is good." It's that we have a reproducibility and infrastructure problem. Google trained these models. Google has the efficient serving code. But getting them to actually run requires reverse-engineering and custom patches. That's not sustainable.

If you're running LLM inference in production, you should care about QAT models. Push your cloud provider or inference framework vendor to make them first-class citizens. Don't settle for post-hoc quantization anymore.

What's your current quantization strategy, and have you tried QAT models yet? I'd love to hear if others are hitting similar friction points.

Source: This post was inspired by "Google's QAT Gemma 4 26B-A4B on One TPU v6e: 15.6x the KV Cache and 1.9x the Throughput of FP8" by Dev.to. Read the original article