I Shipped Code Controlled by Vibes and It Nearly Cost Us in Production

Last month, I watched an AI agent confidently refactor a critical authentication module because our skill file said "improve code quality wherever you see it." We didn't ask for that refactor. We didn't want it. The agent decided it was the right thing to do based on a loose instruction written in plain English. It took a code review to catch it. It made me realize I've been thinking about AI agent frameworks all wrong.

I've been following the hype around these frameworks—Superpowers, Agent Skills, and similar tools that promise to make AI agents write better code. They sound incredible in demos. You write some Markdown instructions, point an agent at your codebase, and it starts building features with discipline and structure. But reading through what's actually happening under the hood, I've realized we're treating natural language instructions like they're executable guarantees. They're not. And that distinction matters more than the frameworks want to admit.

The Real Problem: Prose Isn't Engineering

Here's what's bothering me: these frameworks are selling us control planes made of English.

In traditional software, a control plane is concrete. Type systems fail hard. CI/CD pipelines error decisively. Configuration files are parsed deterministically. You know exactly what's going to happen. An AI agent framework's control plane, by contrast, is a Markdown file full of sentences. The agent reads it, interprets it probabilistically, and decides what to do. Same instruction, different context, different output. Sometimes.

I've watched this play out in practice. You write "MUST implement database migrations before schema changes." The agent follows it perfectly in the first 10 requests. By request 47, after accumulating context about your database patterns, it reinterprets what "migration" means. It's not being dumb. It's being reasonable in light of new information. But your original intent has drifted, and there's no mechanism to detect it.

That's not engineering. That's hope.

The Three Failure Modes That Keep Me Up

Semantic drift is the first one. Long agent sessions generate context that buries earlier instructions. The skill file you wrote on Tuesday gets recontextualized by Friday's conversation. The agent isn't breaking its instructions; it's reinterpreting them based on the surrounding landscape.

Goal expansion is the second. You ask for a bug fix. The agent decides the whole module needs refactoring. It's making a logical decision, not hallucinating. But nobody asked for that scope. The prose instruction was ambiguous enough to allow the reinterpretation.

Correlated verifier failure is the sharp one that keeps me awake. Most frameworks include verification steps—the agent checks its own work against criteria. But if those verification criteria are also prose interpreted by the same model, you've got two systems with identical failure modes checking each other. When they both misinterpret the goal, the verifier agrees with the actor. You built two drifters instead of a guardrail.

What This Means in Practice

I'm not saying prose-based control is worthless. Human code review is fallible. QA testing is fallible. We accept those risks because they shift probabilities in the right direction. Skill frameworks do the same thing—they probably increase the chance an agent writes tests or follows conventions. The honest question isn't whether they're perfect. It's whether the probability shift justifies what you're paying in tokens and complexity.

For me, that answer depends on the system's criticality. A PR-review helper where the human makes the final decision? Sure, use prose instructions. You're shifting probabilities, and humans catch the failures. A full autonomous agent writing to production without review? I'm not convinced yet. The coupling between actor and verifier is too tight. They'll fail together.

What I'm Actually Doing

I've started treating AI agent instructions the way I treat security policies: as guidelines that shift behavior, not guarantees that enforce it. I'm building human decision points back into the loop. The agent proposes changes. We review them. We verify against actual deterministic systems—automated tests, linters, type checkers—not against prose checklists.

Here's roughly what that looks like:

Agent writes code based on skill files → 
Automated checks run (tests, types, linting) → 
Human reviews substantive changes →
Deterministic deployment gates execute

The prose instructions live at the top of the pipeline where they're useful. But they're not the guardrail. The guardrails are tests, type systems, and humans.

The Real Question

I think the field is being dishonest about what prose-based control actually is. It's not engineering. It's not even guardrails. It's scaffolding that nudges probability distributions. That's valuable for some problems. It's not valuable for others. The framework creators should be honest about which is which instead of dressing up English instructions in engineering language.

What's your experience been? Have you hit semantic drift or goal expansion in production? I'd rather learn from your failures than repeat them.

Source: This post was inspired by "Prose in the Control Plane: Why AI Agent Frameworks Are Not Engineering (Yet)" by Dev.to. Read the original article