Stop Hiding Behind "Cloud is Easier": The LLM Architecture Decision I Actually Had to Make

Last month, a client asked me to build a document analysis feature for their compliance platform. They wanted AI-powered categorization of sensitive financial documents. My first instinct? Spin up Claude API, done in an afternoon. Then their legal team said: "Nothing leaves our servers." That's when the cathedral-versus-bazaar question stopped being theoretical for me.

I've been building web apps in Islamabad for years, but suddenly I was staring at a decision framework I'd been avoiding: do we run a local LLM, or do we architect around the constraints of cloud APIs? The answer wasn't obvious, and the surface-level arguments people make online-"local LLMs are too slow" or "cloud APIs are cheaper"-fell apart the moment I did the math for this specific use case.

The False Economy of "Just Use the API"

Here's what I thought before digging deeper: cloud APIs are cheaper because you only pay for what you use. No hardware investment, no DevOps headaches, ship in minutes. This is true if you squint at it the right way.

But the cost comparison is where most analyses go wrong. Yes, Claude runs $2-10 per million tokens depending on the model tier. GPT-4 is pricey. DeepSeek is stupidly cheap. But none of this matters if you don't account for utilization.

The original framework calls this the utilization trap, and I felt it immediately. My client's document processing wasn't steady. Some days: zero documents. Other days: 500. At 10% average utilization of a GPU, the cost per token becomes astronomical compared to running at full capacity. A cloud API charges the same regardless of whether you're sending one token or a thousand. That sounds like a win until you realize you're paying for predictability you don't have.

Privacy Isn't a Feature, It's the Requirement

This was the real driver for my client. HIPAA-adjacent compliance rules, proprietary financial data, the kind of stuff that makes lawyers nervous. Sending data to a third party's infrastructure, even with encryption, isn't acceptable in their world.

I'd always treated privacy as a nice-to-have with cloud APIs-"they're not evil, they have good security." But sitting in a client meeting hearing "this cannot leave our premises," I realized I'd been deflecting a real architectural question with comfortable assumptions.

Local deployment isn't better because it's more private in some absolute sense. It's better because you eliminate an entire dependency chain. No cloud provider audit, no terms-of-service changes, no third-party logging. For regulated industries, healthcare, finance, government, this shifts from preference to requirement.

The Real Math: When Local Actually Wins

I spent a weekend running numbers for an RTX 5090 amortized over 36 months. Hardware, electricity, engineering time to maintain it. The break-even point with mid-tier cloud models (Claude Sonnet range) landed at roughly 135 million tokens per month.

My client's actual usage? Averaging 45 million tokens monthly. That's below break-even, barely. But here's what changed my thinking: that break-even point assumes I stop spending engineering time on infrastructure the moment hardware is amortized. In reality, I'm patching, updating, troubleshooting, optimizing for the next three years. Every unexpected issue costs time I could spend on product.

So I built a hybrid approach. Local LLM for sensitive document analysis that can't leave the network. Cloud API for supplementary tasks where privacy doesn't matter and latency is forgiving. Best of both worlds? More like: managed complexity in exchange for actually solving the constraints.

What I'd Do Differently Next Time

The framework in the original article is solid, but it's missing something I needed: a decision tree that accounts for regulatory reality, not just cost. If you're in a regulated industry, local-first thinking makes sense. If you're a startup optimizing for speed, cloud APIs win. The problem is most of us are somewhere in between.

I also wish there was more discussion about model currency. Open-weight models evolve fast. Today's cutting-edge local model is tomorrow's outdated. Cloud APIs automatically stay current. That's worth something, but how much depends on your tolerance for retraining and redeployment cycles.

The latency argument in the original piece is understated. A local inference server returning results in 50ms versus a cloud API at 200ms doesn't sound dramatic. But in interactive applications, real-time coding suggestions, voice assistants, chat interfaces, that difference is perceptible and matters for UX.

Where I Actually Stand

I'm not evangelizing local LLMs as the future. I'm saying the default assumption that "cloud is better" needs to be questioned per project, per regulatory context, per utilization pattern. Do the math. Talk to your legal team. Measure your actual demand curve, not your peak load.

For my client: hybrid approach, local for sensitive work, cloud for secondary tasks. It's more complex than either pure option, but it's honest about their actual constraints.

What's driving your LLM architecture decisions? Cost, privacy, latency, or something I'm missing?

Source: This post was inspired by "Local vs Hosted LLMs: The Decision Framework" by Dev.to. Read the original article