Three weeks ago, a client asked me if we could turn user prompts into narrated stories for their education platform. My first instinct was to say "that's too complex, too expensive." Then I remembered seeing something about ElevenLabs and realized I could build a proof-of-concept in a few hours. Spoiler: I did, it worked, and it changed how I think about combining APIs in production. But not everything in the obvious path is worth taking.
The appeal is immediate, users type a prompt, get back a fully narrated story with different character voices. It feels like magic. It is magic, technically. But sitting down to actually build it forced me to think hard about what's worth the complexity and what's just nice-to-have theater.
The Pipeline Is Simpler Than You'd Think
Here's what actually happens: your frontend sends a prompt to a backend that asks an LLM (GPT in this case) to structure it as a script with speaker tags. Then each line gets sent to ElevenLabs' text-to-speech API, which returns an MP3. The frontend stitches them together into a playlist. Done.
The architecture is beautifully boring, which is exactly what you want. The hard work, natural language understanding and voice synthesis, lives on the server side. Your JavaScript doesn't need to think. It just collects URLs and plays them in order.
What I appreciated about this approach: it keeps the client lightweight. No heavy models running in the browser. No waiting for the user's connection to download gigabytes of stuff. The tradeoff is you're making multiple API calls, which means latency and costs. More on that later.
Where I'd Push Back
The original article makes this sound like a weekend project that "just works," and technically it is. But there are real production concerns buried under the tutorial code that need addressing.
First, the economics don't scale gracefully. ElevenLabs charges by character, not by request. A ten-line story might cost you a few cents. Multiply that by thousands of users, and you're looking at real money. The article mentions an affiliate link but never talks about cost management or caching strategies. I'd immediately implement Redis caching, if someone generates "A brave rabbit discovers a hidden city" twice, serve the cached audio the second time.
Second, the error handling in that Flask code is basically nonexistent. What happens when OpenAI times out? When ElevenLabs rejects a request because the API is rate-limited? When your temporary file storage runs out of space? In production, you need retry logic with exponential backoff, queue systems, and fallback voices.
Third, and this bothers me more than it should, voice cloning requires uploading actual voice samples to ElevenLabs. That's data you're storing with a third party. If your story generator is for kids, or uses real voices without permission, you're playing with legal fire. The article glosses over this entirely.
What I'd Actually Build Differently
I'd separate concerns more aggressively. Instead of synchronously generating audio in response to a user request, I'd queue the job, generate audio in the background, and notify the user when it's ready. This means better UX (no 10-second hang) and cheaper infrastructure (you can batch API calls).
I'd also invest in prompt engineering upfront. The magic isn't ElevenLabs, that's commodity TTS now. The magic is getting GPT to consistently produce good scripts. With the right system prompt, you can control story length, pacing, dialogue patterns, everything. I spent probably 40% of my implementation time tweaking that.
Here's a simplified version of how I handle the backend:
@app.route("/api/story", methods=["POST"])
def create_story():
data = request.json
prompt = data.get("prompt", "")
job_id = str(uuid.uuid4())
# Queue the job instead of blocking
queue.enqueue(generate_and_synthesize, job_id, prompt)
return jsonify({"job_id": job_id, "status": "processing"})
@app.route("/api/story/<job_id>", methods=["GET"])
def get_story_status(job_id):
result = cache.get(f"story:{job_id}")
if result:
return jsonify({"status": "complete", "audio_urls": result})
return jsonify({"status": "processing"}), 202
This way, the frontend polls for completion instead of hanging. Real-world improvement.
The Real Question
Here's what I'm wrestling with: should every interactive experience be audio-enhanced? The article doesn't ask that. It just assumes audio is additive value. But I've seen projects where adding narration actually hurt engagement, users found it intrusive, wanted to read at their own pace, or felt weird having AI voices read to them.
Before you implement this, ask yourself: Why audio? Is it accessibility? Engagement? Novelty? Each answer leads to different implementation choices.
Source: This post was inspired by "Create an Interactive Audio Story Generator" by Dev.to. Read the original article
