I got a message from our finance team last week asking why one batch summarization job cost 3x more than another, even though they processed similar data. I couldn't answer them without digging through logs for twenty minutes. That's when I realized we've built the entire LLM integration without any real cost visibility, and worse, no way to actually charge back to the right customer.
This happens a lot in startups building on top of LLMs. You fire off batch jobs, they work, results come back, everyone's happy. Nobody asks the hard question: who pays for this? And when you do ask it, the answer is usually a shrug and a spreadsheet.
I spent yesterday reading through a detailed breakdown of how to actually structure this properly, and it forced me to admit we've been doing it wrong.
The Batch vs. Realtime False Choice
Here's what I thought before: batch jobs are cheaper, realtime is faster, pick one. That's surface level.
The real decision is about who's waiting. If a customer asks "why was my content rejected?" they need an answer now, not tomorrow morning. That's realtime. But when we're running nightly summarization on 50,000 marketplace reviews? Nobody's sitting there waiting. That's batch.
The mistake I see teams make is treating this as purely a cost optimization. It's not. It's a deadline decision first. You can't retroactively decide to batch something when a seller is angry about a listing decision. You batch the work that has flexibility, and you keep realtime calls for the interactive path.
But, and this matters, batch doesn't magically solve the latency problem. It just moves it to a different place in your workflow. You still need to poll, validate, and eventually surface results to users. That happens to look different, but the total time from "input" to "actionable output" is usually not that much faster.
Cost Attribution Is the Real Problem
This is the part that made me go back and reread multiple times: cost attribution without proper accounting is a disaster waiting to happen.
When you send a batch job to an LLM provider, it's a black box from the outside. Everything goes in, everything comes out, one provider bill. But internally? That job probably contained data from five different customers. How much did each one cost?
The approach that stuck with me: create an immutable ledger record before you send anything to the provider. Parent row per tenant batch, child rows per item. Track estimated tokens upfront, actual tokens after completion, and distribute the provider's final bill according to a documented rule.
This means storing estimated input tokens, reconciling against actual provider metadata, and having a clear record you can show to finance or a customer. Not guessing. Not assuming cost spreads evenly. Knowing.
The compliance angle is also critical: you can't let cost visibility become data leakage. Tenant A shouldn't see that Tenant B paid less per unit. The ledger is internal. The retention policy is intentional.
State Machines Matter More Than You Think
I've written enough job queue systems to know that "waiting for completion" is amateur thinking. The original article breaks down actual states: planned, submitted, running, results_ready, validated, applied. Each one is distinct.
This matters because polling can repeat, workers can restart, and result exports can be read twice. If your state machine isn't explicit about this, you'll double-charge or double-process without even realizing it.
Building idempotency into the apply step isn't optional. It's the difference between a system that breaks silently and one that handles reality.
My Take
I agree with almost all of this. The one place I'd push back slightly: the Python snippet they included is useful, but the bigger lesson is that you need observability in your own system, not just at the provider boundary. A 429 from the provider tells you something. But you also need to know: how long did this job actually sit in your queue before submission? Did validation reject anything? How many retries happened?
The batch architecture they describe is boring. That's the point. I spent too many years trying to be clever with async systems, and boring ones are the ones that survive production.
What I'm Doing Monday
I'm auditing our current LLM integration. We're probably burning money we don't know about, and we're definitely not able to show customers what they actually cost us. That needs to change before we add another feature on top of it.
What about your team? If you're processing LLM jobs at any real scale, do you know what they actually cost per customer? Can you explain it to finance?
Source: This post was inspired by "Batch LLM Jobs vs Realtime APIs, Bulk Summarization Cost Attribution" by Dev.to. Read the original article
