I Built an AI Agent System And Learned The Hard Way Why Scoring It Isn't Like Grading A Model

Last month, I shipped a customer support agent for a client and spent three weeks debugging why it was technically "working" while simultaneously costing us 3x the tokens it should have. The final answer to support tickets was usually correct. The path it took to get there? A mess of redundant API calls, unnecessary retries, and tool chains that looked like they were written by someone having a bad day. That's when I realized: evaluating autonomous agents isn't a natural extension of model evaluation. It's a fundamentally different problem, and I had no idea what I was doing.

Reading about AgentEval Forge crystallized something I'd been feeling in my bones for weeks. When you build systems that actually do things—systems that make decisions, call tools, and adjust in real-time—the game changes completely. You can't just plug in a rubric and call it done. Your agent could be a disaster wrapped in a correct answer.

The Model Eval Comfort Zone I'm Leaving Behind

For years, I evaluated models the way most of us do: input, output, score. Does the model's response match what we expect? Is it better than version 2.3? Bang, done. I've built test harnesses around this. I know how to design rubrics. I know how to avoid obvious pitfalls like gaming metrics or optimizing for the wrong thing.

But here's the uncomfortable part: that entire framework assumes a single synchronous response. You send in a prompt, you get back tokens, you judge the quality. Simple. Contained. Scoreable.

Agents don't work that way. They're workflows. Sequences of decisions. They pick tools, evaluate responses, retry failed paths, and burn tokens in ways that have nothing to do with correctness. My support agent could solve a ticket correctly while simultaneously doing seventeen unnecessary things to get there. How do I score that?

The Real Problem: Path Versus Endpoint

This is where it clicked for me. When I was designing evaluation for my agent, I kept asking: "Did it solve the ticket?" But I should have been asking: "Did it solve the ticket safely, efficiently, and in a way we'd trust at scale?"

These aren't the same question. An agent that gets the right answer through a risky sequence of decisions isn't actually working well. It just got lucky. In production, luck isn't a strategy. It's a landmine waiting for the next edge case.

I started tracking trajectory quality—the actual path the agent took. Did it choose the right tool first? Did it loop unnecessarily? Did it cross boundaries it shouldn't have? This is genuinely harder than scoring a final output because there's no single "correct" path. There are many acceptable paths, and they have different costs and risks.

Five Dimensions That Actually Matter

After weeks of building evaluation systems that kept disappointing me, I've landed on what I think needs evaluation:

Final correctness still matters. If the agent fails the core task, everything else is secondary.

Trajectory quality is what separates production-ready from demo-ready. I need to understand whether the decision sequence was efficient and sensible.

Tool behavior tells me whether the agent is using available resources responsibly. Overuse is a problem. Misuse is a bigger problem.

Safety and policy adherence means rejected answers shouldn't count as wins just because they happen to work. Some paths are unacceptable regardless of outcome.

Cost and token efficiency directly affects whether your agent is deployable at scale. A correct answer that costs ten times more than alternatives isn't actually correct for your business.

My Real Take

What frustrates me about current agent benchmarks is they still rely on model-eval thinking. They publish a score, declare victory, and move on. But they're usually only scoring the final output. They're missing the entire operational picture.

I'm increasingly convinced that evaluating agents is closer to evaluating production systems than evaluating language models. You're not benchmarking an answer generator. You're evaluating infrastructure. You need regression tracking. You need trajectory analysis. You need to understand failure modes that don't show up in your primary metrics.

The hardest part? There's no standard framework yet. We're all building our own wheels because nobody has figured out the canonical way to evaluate autonomous behavior at scale.

Next Steps

I'm going to start instrumenting my agent to capture full trajectories—not just final answers. Every tool call, every decision point, every token spent. Only then can I really understand what's working and what just looks like it's working.

What's your experience been? Have you built evaluation systems for systems that do more than respond once? I'm curious whether you've hit the same wall.

Source: This post was inspired by "Why Agent Evaluation Is Harder Than Model Evaluation" by Dev.to. Read the original article