Last month, a client's CRM integration started costing us three times what we quoted. I spent an afternoon digging through API bills only to realize we were sending entire sales call transcripts, complete with dead air, tangents, and repeated greetings, to an LLM endpoint on every update. That's when I understood something I'd been glossing over: every token you send to an LLM has a price tag, and the cost compounds when you're processing data at scale.
I'd been thinking about LLM integration like I think about database queries, get the right answer, optimize later. But token economics work differently. When you're processing hundreds of call recordings daily, "later" arrives at invoice time, and by then you've already burned money on context that didn't matter.
The Real Problem Isn't Transcripts, It's Waste
Here's what I missed: the cheapest-looking API call isn't always the cheapest operation. A long transcript sent through an LLM costs money on both the input and output. Retries multiply that cost. But more importantly, extra context doesn't always improve results, it often just pads the bill.
The original article nails something I've been learning the hard way: the goal isn't shorter transcripts at any cost. The goal is the smallest evidence window that preserves what actually matters, the customer's need, who owns the next step, what was committed to, what happens next. Everything else is noise.
In my CRM implementation, I was sending speaker labels, timestamps, and entire opening conversations. None of it affected whether the LLM extracted the right owner or due date. Cutting it would've saved tokens. But I didn't cut it because I wasn't measuring it.
Measure Before You Optimize
This is where the article's approach clicked for me: store the original call, record the selected evidence, track input token count, capture output count, and evaluate the result against a fixed rubric. All in one evaluation record.
I started building this into my Python workers. I set a 12,000-character guard as a preprocessing boundary, not because it's a magic number, but because it's visible, trackable, and forces me to be intentional about what I'm passing downstream.
The rubric matters more than the count. A lower token count is a fail if the model drops the due date or misses the account owner. I've seen that happen. The semantic search returned the wrong passage. The reranker picked something that sounded relevant but wasn't actionable.
Smart Retrieval Over Raw Context
This is where I'm experimenting now. Instead of passing the entire call, I'm using retrieval to extract targeted evidence: separate queries for customer need, commitments, ownership, and open questions. Then I merge a bounded set of relevant passages.
Embeddings work well for this because the LLM gets evidence instead of noise. But retrieval creates its own failure modes, the highest-scoring passage might mention an objection while the actual owner and date sit in the next result. That's why I'm building eval sets with hard examples: long calls, repeated names, corrections, commitments buried at the end.
My Take: This Changes How I Build
I agree with the article's core argument, but I'd push it further: token optimization isn't a cost-reduction tactic. It's a design constraint that forces you to think clearly about what your LLM actually needs to do.
Before, I'd build a feature, then ask "how much will this cost?" Now I'm asking "what's the minimal contract this needs?" and working backward. That shift has changed how I design prompts, truncate transcripts, and structure data.
Here's a practical example I'm using:
from dataclasses import dataclass
@dataclass(frozen=True)
class EvalRecord:
original_call: str
selected_evidence: str
input_token_count: int
output_token_count: int
extracted_action: dict
truncation_occurred: bool
retrieval_recall: float
pass_rubric: bool
# Store this alongside every LLM call.
# Query it to find which evidence windows work best.
The goal is a feedback loop. Run enough calls, score the results, analyze which evidence selections consistently produce correct extractions. Then standardize on that pattern.
The Question I'm Still Sitting With
The article assumes you have control over the CRM output contract and can keep it small. But what if the next feature request wants more fields? What if stakeholders push for richer context? Token discipline is easy until business requirements arrive.
I'm curious how others handle that trade-off. Have you found a way to push back on scope creep when it directly impacts LLM costs?
Source: This post was inspired by "Python LLM API Token Count Impact on CRM Cost and Usage Reduction" by Dev.to. Read the original article
