Stop Building RAG Search Like It's a Magic Bullet, Start With What Actually Fails

I spent three weeks last year watching our support team get increasingly frustrated with our "smart" documentation search. We'd invested heavily in vector embeddings, hired someone to fine-tune a retrieval model, and deployed the whole thing with genuine confidence. Then a customer couldn't find how to reset their API key because they typed "regenerate credentials" and our semantic search thought that was close enough to "create new authentication token." Meanwhile, the exact phrase they needed was buried on page seven.

That's when I realized we'd solved the wrong problem first. We'd optimized for sophistication instead of correctness. I've been thinking about that failure ever since, and reading through a recent discussion about ask-your-docs architecture made me realize just how much we got backward.

The Honest Truth About Embeddings vs. Keyword Search

Here's what I think most developers get wrong: we treat this like a choice. Embeddings or keyword search. Binary. But that's not how real support queries work.

Semantic embeddings solve a genuine problem, the language mismatch between how customers describe their issue and how your documentation phrases the solution. Someone asks "how do I change who owns my store" and your docs say "transfer account administration." Without embeddings, you're dead. The query tokens don't overlap. Embeddings map both into the same semantic space and actually retrieve something useful.

But embeddings are terrible at exact matching. If a customer pastes an error code like PAYMENT_1042, or asks about a specific API field name, or searches for their plan tier, embedding that query is pure noise. You need literal token matching for those cases. And you need it to win.

The architecture that stuck with me from the article was refreshingly pragmatic: start with embeddings as your primary retrieval layer, then explicitly check whether the query contains identifiers that need exact matches. If it does, merge in those results. No complex multi-stage ranking. No premature optimization. Just: does this look like something that needs to match exactly, or something that needs semantic understanding?

Where the Real Problem Actually Hides

But here's what made me lean back in my chair: the billing and observability angle. Every SaaS company I've talked to has made the same mistake we did. They optimize the search quality in isolation and then get blindsided by a bill they can't explain.

The article makes this point that feels obvious once stated: if you're calling a vector database, a reranking model, and an LLM to answer questions, and you're only logging aggregate tokens at the final step, you've essentially created a black box where you can't attribute costs to tenants. You know your total bill. You have no idea which customer created it.

This happens because teams treat these calls as application internals rather than billable operations. Each call to your retrieval system, your reranking service, your model, all of it should emit an immutable event tied to a tenant ID before it leaves your service boundary. Not aggregated. Not sampled. Individual events that you can join against product usage later.

We're doing this wrong in production right now, and I know it.

What I'd Actually Build Today

If I were starting an ask-your-docs feature from scratch today, I'd do this:

First, store chunked documents with their tenant ID embedded as an invariant, not a filter. The chunk should never exist in the system without tenant context. This sounds obvious until you're debugging why customer A saw customer B's documentation.

Second, retrieve a small candidate set with vector similarity, scoped to the tenant. Keep it boring. The request path should be: authenticate → resolve tenant → vector search → optional rerank → send to LLM. No cleverness.

Third, emit a usage event for every external call. Every single one. Include the tenant, the request ID, the vendor response, latency, cache status. Store that before you acknowledge the user's request, because if the user closes their browser, you still need that event.

Fourth, actually test whether reranking helps. Don't deploy it because it sounds smart. Deploy it when your evaluation shows the first-stage vector retrieval is legitimately weak. The article emphasizes this and I think it's right, reranking is a second-order optimization. Fix missing documents and bad chunking first.

The Question I'm Still Sitting With

Here's what I don't have a clean answer for yet: how do you decide ownership of cost when a request fails? If a retrieval times out, is that a platform cost or a tenant cost? The article mentions this briefly but doesn't prescribe the answer. I think the invariant is right, keep the raw event auditable, but the policy needs to be explicit before you build it.

Source: This post was inspired by "Ask-Docs Architecture: Semantic Embeddings or Keyword Search for a SaaS Help Center?" by Dev.to. Read the original article