Last month, I spent a weekend setting up Qwen locally because I was tired of Claude API bills. I had a 24GB GPU gathering dust, I'd seen the SWE-bench scores, and the math seemed obvious. By Sunday afternoon, I had a working setup. By Monday morning, I realized I'd completely misunderstood what "working" actually meant.
The model could write functions. Beautiful functions. Functions that passed tests. But the moment I asked it to refactor three files while maintaining a specific pattern, it started drifting. The tool calls became sloppy. It stopped respecting the constraints I'd set. After a week of tweaking, I went back to Claude. I wasn't angry about it, I was just surprised that nobody in those setup guides mentions the gap between "this model scored well" and "this model is reliable enough to ship."
That's the core insight I got from rereading everything about local LLMs in late 2025 and early 2026. The benchmarks are measuring the wrong thing entirely.
The Benchmark Problem Nobody Wants to Admit
Here's what SWE-bench Verified actually tests: given a bug description, produce a patch. That's one-shot writing. It's clean, measurable, and it makes for great headlines. But when I'm using Claude Code, or trying to use a local model, I'm not asking for one patch. I'm orchestrating a sequence of edits across multiple files while the model tracks context, respects constraints, and doesn't hallucinate tool calls.
These are completely different problems. SWE-bench is like measuring a chef's ability to cook a single perfect steak. It says nothing about whether they can run a kitchen service, plate dishes while staying in rhythm with the line, or remember that table six ordered medium-rare, not well-done.
A local LLM writing correct code in isolation is solved. That's not the question anymore. The real question, the one that decides whether I close Claude's tab, is whether it can be a reliable agent over a sustained interaction.
What Actually Breaks When You Go Local
I've identified four failure modes that don't show up on any leaderboard, but they're the reason most self-hosted setups get abandoned after two weeks:
Tool-call validity is the first. The model needs to emit well-formed edit commands that my harness can parse reliably. Not usually. Reliably. When it starts hallucinating parameters or malforming JSON, the whole interaction collapses.
Long-horizon consistency is brutal. By the tenth tool call, does the model still remember what it said in call two? I've watched local models forget their own multi-file strategy mid-session. Claude doesn't do this.
Context under load matters because real repositories aren't 5KB of toy code. When I load 40K tokens of actual context, do I get the same quality? Most open models degrade gracefully but perceptibly. Some collapse like a house of cards.
Instruction-following is the unsung one. When I say "don't modify the test files," does it listen? Or does it optimize locally and ignore my constraints? This sounds trivial until you're debugging why your local model broke your test suite.
The Actual Model Families That Are Worth Your Time
After digging into this, I've narrowed the field down. Qwen is the obvious pick if you're serious, 27B dense or 35B MoE, depending on your hardware. It does tool-calling and coding reasonably well and doesn't disappoint when you ask it to hold context over multiple interactions.
GLM-4 is the dark horse for laptop-tier hardware, especially if you're running long agentic sessions. It's lighter and surprisingly consistent over many tool calls, which matters more than raw single-shot performance.
DeepSeek V4-Pro scores highest on paper but requires hardware I'll never own. That's an escalation option, not a daily driver.
What I won't do anymore is buy GPU capacity based on SWE-bench percentages. That number is necessary but not sufficient. I need to know: does it reliably emit valid tool calls? Does it hold strategy over ten interactions? Does it respect my constraints? Those aren't on the leaderboard, and they should be.
My Actual Next Step
I'm building a small eval suite tailored to agent behavior, not coding correctness, but tool-call reliability, instruction following, and context consistency over 30+ interactions. I'm going to run it against the models I'm considering and actually measure what decides whether I use it daily.
The setup is fast now, the hardware is achievable, and for many workloads, local inference makes economic sense. But the benchmarks need to catch up to the actual job. Until they do, go in with your eyes open about what you're comparing.
What's your experience been? Have you tried running a local setup and found the gap between the scores and the reality? I'd genuinely like to hear where you hit walls.
Source: This post was inspired by "Can a local LLM replace Claude Code? The benchmark everyone quotes is the wrong one." by Dev.to. Read the original article
