I Built a RAG System for My Client. Then Production Happened.

Six months ago, I pitched my client on a slick RAG architecture. Index their documentation, connect it to an LLM, done. They'd have an AI assistant that could answer questions about their policies instantly. I was confident. I'd read all the right articles. I knew the theory.

Two weeks into production, that confidence evaporated.

The first user asked the system something innocent: "Can I process this customer refund?" The RAG system retrieved the refund policy perfectly. The LLM explained it clearly. Then the user asked me why the system didn't check if they actually had the permission to approve refunds, or whether the customer's account was in good standing, or if there were regional restrictions that applied. I realized I'd built a beautiful knowledge retrieval system masquerading as a decision-making system. That's when I understood what everyone keeps getting wrong about enterprise AI.

The Problem Isn't RAG Itself

Let me be clear: RAG solved a real problem. Before RAG, the only way to make an LLM aware of your internal knowledge was to retrain it every time something changed, expensive, slow, and impractical. RAG gave us a clean solution: retrieve relevant context at runtime and let the model work with it.

The issue is that we've started treating RAG like it's the complete answer to enterprise AI. It's not. It's one piece of a much larger puzzle. In my client's case, I was so focused on the retrieval layer that I ignored everything else the system actually needed to do its job.

Where RAG Breaks Down

The moment your AI system has to make a decision or take an action, retrieval stops being enough. It's the difference between "What is our policy?" and "Should I approve this refund?" The first is a retrieval problem. The second is a software engineering problem.

Think about what needs to happen. You need to verify the user's identity. Check their role and permissions. Pull real-time data from multiple systems. Apply business rules. Handle edge cases and exceptions. Maintain state across multiple steps. Ensure everything is auditable. Sometimes get human approval. Then actually execute the action.

A vector database can't do any of that. And frankly, the LLM shouldn't be responsible for figuring it out on its own.

The Real Architecture Is Much More Complex

What I've learned is that production enterprise AI looks less like "retrieval + generation" and more like "orchestration + reasoning + action." The retrieval part is still there, but it's embedded inside a much richer system.

You need authentication and authorization layers. You need integrations with actual business systems, APIs, databases, payment processors, ticketing systems. You need to track state. You need validation at multiple checkpoints. You need observability so you can debug what went wrong. And you need governance, rules about when the system can act independently and when it needs human intervention.

Building this is much closer to building a traditional distributed system than it is to prompt engineering.

What I Actually Built

For my client, I ended up creating a workflow engine that used the LLM as a reasoning component, not the primary component. Here's the flow:

1. Parse the request - Understand what the user is trying to do
2. Retrieve context - Pull relevant policies and documentation (this is the RAG part)
3. Gather state - Query live systems for user permissions, customer data, current approvals
4. Reason - Let the LLM synthesize all this information and propose an action
5. Validate - Check business rules and authorization
6. Execute or escalate - Either perform the action or route it to a human

The LLM is powerful, but it's one component in a larger system. And that system has to be built like any other critical software system: with tests, monitoring, error handling, and careful state management.

The Question That's Bothering Me

I keep wondering if we're setting ourselves up for failure by marketing RAG as "enterprise AI." RAG is a retrieval pattern. It's genuinely useful. But calling it "enterprise AI" is like calling a database query engine a "business system." They're components, not solutions.

My real question: Are we building the right mental model for how AI actually gets deployed in production? Because if you're still thinking of this as a retrieval + LLM problem, you're probably not thinking about the hard parts yet.

What's your experience been? If you've built something beyond basic RAG, what did you have to add to make it actually work in production?

Source: This post was inspired by "RAG Is Not Enough: The Evolution of Enterprise AI" by Dev.to. Read the original article