I Finally Did the Math on Self-Hosted AI, and It Changed What I'll Recommend to Clients

Last month, a client asked me the question that's haunting every development team right now: "Can we switch to open-source AI models and cut our API bills in half?" I nodded and said I'd look into it. Honestly, I was expecting to come back with a "technically possible but not worth the headache" answer. Instead, I spent two weeks reading pricing pages and playing with quantized model deployments. The numbers were surprising enough that I had to actually think through this rather than dismiss it.

What started as a quick research sprint turned into me deploying a 7B model on EC2 just to see what happens. And now I'm convinced that self-hosted open models are hitting a real inflection point—not because they're always cheaper, but because for the first time, the operational complexity is low enough that the math actually works for mid-sized teams.

The Thing That Clicked For Me: GPU Memory Isn't About Speed, It's About Parallelism

The original article nails something that I kept misunderstanding until I actually ran the numbers myself: GPU memory isn't a speed thing. It's a throughput thing.

A 7B model running on a CPU gets you 2-5 tokens per second. On a GPU, it's 30-80. For one person, that's fine. But for a team of ten queuing requests? It's dead on arrival. The highway metaphor in the article stuck with me—a CPU isn't a slower car, it's a single-lane road. GPUs are thousands of lanes.

I tested this locally before I went anywhere near AWS. Deployed Llama 2 with vLLM on my development machine with an RTX 3060. Watched concurrent requests actually stay responsive. Then I capped it at CPU-only inference. Immediate regret. Everyone was waiting 45 seconds for responses that should've taken 3 seconds.

That's the moment I realized this wasn't theoretical anymore. The problem isn't computational power—modern CPUs have that. The problem is that language model inference isn't about doing one calculation fast. It's about doing billions of calculations in parallel for dozens of requests at once. That's just what GPUs are built for.

The Real Cost Picture (And What It Actually Means)

Here's where I stopped nodding along and started thinking carefully. For a team of ten with standard working hours, running a single g5.12xlarge with four A10G GPUs costs roughly $1,250/month if you stop the instance at night. That's less than most teams spend on a mid-tier ChatGPT Enterprise subscription.

At 500 people, you're looking at $9,000-12,000/month for a solid setup with good response times. That's competitive with what you'd pay hitting the Claude or GPT-4 APIs for moderate usage. But here's what changes the equation: you own the data. No API logs. No request tracking. That matters for regulated industries or companies building proprietary workflows on top of the model.

What the article doesn't emphasize enough, though—and I learned this the hard way—is the operational tax. Someone has to keep this running. Model updates. Instance reboots. Monitoring. Scaling adjustments. For ten people, that's probably not justified. You'd actually save money just paying per-request fees and letting Anthropic or OpenAI deal with the infrastructure.

For 500 people? Now you're paying someone (or a small team) to manage this, and the math still works. You've got enough volume that the infrastructure becomes economical.

What I'd Actually Do Right Now

If a client with 50-200 people asked me today, here's my honest answer: Start with vLLM and an open model. The software stack is genuinely mature. You get an OpenAI-compatible API, which means zero code changes if you ever switch back. Set up basic monitoring with CloudWatch. Use spot instances where possible (they'll save 50-70% on GPU costs, though you'll need to rebuild if they terminate).

But I wouldn't go all-in on self-hosting frontier-class models. The infrastructure complexity for Kimi K3 or the 400B parameter models is real. You need serious monitoring. You need redundancy. That's a different engineering project.

What I would do is start with a smaller model—Llama 4 Maverick or Qwen 3.5—deploy it on a single instance, and actually measure your usage patterns. Most teams dramatically overestimate how much inference they need.

The Question I Can't Stop Thinking About

If open models are genuinely this cost-competitive now, why aren't more teams doing this? I think the answer is organizational friction, not technical friction. It's easier to put a ChatGPT API key in an environment variable than to argue about infrastructure complexity in your next sprint planning meeting.

But that inertia is breaking. I'd watch this space closely.

Source: This post was inspired by "How Much Does It Cost to Self-Host Open Models on AWS?" by Dev.to. Read the original article