The Quantization Problem I Didn't Know I Had Until I Tried Running LLMs Locally
I spent three weeks last month trying to serve a 26B parameter model on a single GPU in my test environment. I kept hitting the same wall: the memory just wouldn't cooperate. I'd quantize with one method, get mediocre results. Try another, burn tokens on latency. I was ready to g...