I Built an LLM Feature Without Token Limits. It Cost Me $900 in Two Days.

Last month, I deployed a customer support chatbot that seemed innocuous enough. Simple retrieval, straightforward responses, nothing fancy. By day two, the AWS bill had spiked by nearly a grand. I spent an hour debugging before I realized the problem: I wasn't limiting conversation history, and support conversations were running 30+ turns. Every single turn was sending the entire chat history back to the API. That's when I learned that building LLM features isn't about raw capability—it's about understanding where tokens actually go.

Most developers approach AI features like we approach regular APIs. You send a request, get a response, move on. But LLMs don't work that way. The economics are hidden in the mechanics. You're not paying for computation; you're paying for tokens processed, and some tokens cost way more than others. Understanding this isn't just about saving money—it's about building features that actually scale.

The Output Token Problem Nobody Talks About

Here's the thing that surprised me: output tokens cost five to eight times more than input tokens. On OpenAI's pricing, you're looking at $1.25 per million input tokens but $10 per million output tokens. That's not a typo.

And reasoning models make this worse. When you use a model with extended thinking, it generates hidden reasoning tokens that get billed at the same expensive output rate. You might get 200 tokens the user sees, but the model burned 2,000 thinking tokens to get there. That's 2,200 tokens you're paying for. On a busy endpoint, reasoning output can dwarf your entire input cost.

This is why I now always set maxTokens() in every Spring AI call. Not as a quality lever—a truncated response is still billed in full—but as a safety valve. Pair it with a prompt that explicitly asks for brevity, and you've got real control.

The Conversation History Trap

Here's where I made my mistake. In a 50-turn conversation, the unbounded approach sends roughly 333,750 input tokens across all turns. With a 10-message sliding window? 86,250 tokens. That's a quarter of the cost.

The math is brutal. Every turn you send carries the weight of every previous turn. Turn 6 onwards plateaus at a fixed size, but the cumulative cost across a session explodes exponentially. For support scenarios where users expect long conversations, this is your biggest lever.

Spring AI gives you MessageWindowChatMemory with configurable window sizes. I use this now on every memory-enabled client.

My Take: Three Things I'd Do Differently

First, I'd be explicit about window sizes in code comments. The default changed from 500 to 4096 tokens in Spring AI 2.0—if you didn't notice, your costs just went up 8x after an upgrade. That's a foot-gun.

Second, I'd test locally with billing models in mind. Ollama models like qwen3 and deepseek-r1 use reasoning by default, but you don't see the token bill locally. I now explicitly disable reasoning in development to catch these patterns before production.

Third, I'd actually measure what's happening. Add metrics around token usage by feature, by user, by conversation length. The article focuses on controls, but you need observability to know if those controls are working. I added token counting to my ChatClient wrapper months ago—best decision I made.

The Code That Actually Matters

@Bean
ChatClient chatClient(ChatClient.Builder builder, 
                      ChatMemoryRepository repository) {
    ChatMemory memory = MessageWindowChatMemory.builder()
        .chatMemoryRepository(repository)
        .maxMessages(10)  // Explicit window - 5 exchanges max
        .build();
    
    return builder
        .defaultAdvisors(
            MessageChatMemoryAdvisor.builder(memory).build()
        )
        .build();
}

This is straightforward, but the impact is massive. A 10-message window keeps conversations reasonable while capping your cumulative cost per user. For my support bot, this single change dropped costs by 75%.

What's Your Token Strategy?

The gap between building LLM features and building them economically is wider than most of us realize. You can't ignore token accounting at scale—it compounds too fast.

Are you measuring token usage in your LLM features today? Or did you learn about this the expensive way like I did?

Source: This post was inspired by "Spring AI Prompt Caching and Chat Memory: Where the Tokens Go — LLM Cost Control 2/4" by Dev.to. Read the original article