DevOps & Cloud

The Question Nobody Asks Until Production Breaks: Who Actually Deploys This Thing?

A

Admin User

Author

Aug 4, 2026
4 min read
2 views
The Question Nobody Asks Until Production Breaks: Who Actually Deploys This Thing?

I had a small panic attack at 2 AM last month. A payment processing service I'd built was running perfectly in staging, tests passed, the Docker image was sitting in our registry looking clean and ready. Then someone asked the question that made my stomach drop: "Okay, how do we actually put this live without losing money?"

I realized I'd spent weeks optimizing build times and container configurations without ever having a real conversation about how the switch gets flipped. I'd been thinking like a builder, not like someone responsible for keeping the lights on. That distinction matters more than I initially understood.

The Simplest Approach is Also the Most Dangerous One

Let me be blunt: I've seen production deployments that just stop the old container and start the new one. Sometimes it works fine. Sometimes it doesn't, and when it doesn't, the consequences depend entirely on what your service actually does. If you're running an internal admin dashboard? Fine. If you're handling transactions? That gap between "old stopped" and "new started" is now a direct line to unhappy customers and, worse, lost trust.

The article calls this the "recreate" pattern, and there's nothing inherently wrong with it—but using it accidentally is a disaster. The real question you need to answer first is whether your service can afford even a few seconds of downtime. Most production services cannot.

Blue-Green: The Expensive Security Blanket

Blue-green deployment is the approach I think every team should at least understand, even if they don't end up using it. You run two full environments simultaneously. One is live ("blue"), one is idle ("green"). When you deploy, the new version starts on green. You can test it, watch it, verify it's actually working—all while real users continue hitting blue without interruption.

Then you flip the router. Traffic switches instantly. If something goes wrong with green once it's live, you flip back immediately. The old version never stopped running.

The tradeoff is infrastructure cost. You're essentially paying to keep a full backup environment idle most of the time. For a small team or tight budget, that's genuinely expensive. But if you're running something people depend on financially, it's often worth it.

Rolling Deployments: The Practical Middle Ground

Most teams I know use rolling deployments, and honestly, that's what I've gravitated toward too. Instead of an all-or-nothing switch, you replace instances gradually—maybe two at a time across a fleet of ten. For a few minutes, both the old and new versions are handling real traffic simultaneously.

Here's what keeps me up at night about rolling deployments: they expose a category of bugs that don't exist in staging. What happens when the new version writes data that the old version can't read? What if they disagree about API formats? You'll discover this in production, mid-rollout, with real users hitting both versions. I've seen reports like "weird intermittent failures for exactly ninety seconds" and that's usually what's happening.

The hard question rolling deployments force you to answer: can these two versions actually coexist safely? You can't just assume yes.

Canary: Testing With Your Real Users (Carefully)

Canary deployments are what I'm most interested in right now. Instead of committing to a rollout, you route a small percentage of traffic—maybe 5%—to the new version while everyone else stays on the old one. You watch metrics specifically for that slice: error rates, latency, exceptions. If something's wrong, those 5% of users hit it, not everyone.

The psychological difference is huge. You're not making a binary decision to deploy. You're making a series of small decisions: is 5% healthy? Now try 25%. Now try 100%. You can stop at any point and roll back without affecting the majority of your users.

What This Means for How I Deploy Now

I've started asking these questions before I even write deployment code:

  • Can we afford downtime? Even 30 seconds?
  • How much infrastructure budget do we have?
  • How confident am I that the new version won't break mid-rollout?
  • If something goes wrong, what's my fastest path back?

The answer to those questions determines everything—blue-green, rolling, or even recreate. The mistake isn't picking the "wrong" strategy. It's not picking one at all and hoping for the best.

What deployment strategy are you using right now, and more importantly—did you choose it deliberately, or did it just happen?


Source: This post was inspired by "Episode 5 — Who Gets to Flip the Switch" by Dev.to. Read the original article

Share this article

Written by Adil Sher

Full stack developer building high-traffic platforms, AI services, and custom web applications. Explore my portfolio, learn about my background, or get in touch.

Related Articles

Quantum Cryptography Broke My Messaging Pipeline (And Yours Might Be Next)
DevOps & Cloud Aug 3

Quantum Cryptography Broke My Messaging Pipeline (And Yours Might Be Next)

I had an incident last month that I've been turning over in my head ever since. One of our microservices started timing out on message processing—nothing catastrophic, but enough to trigger alerts. After digging through logs, I found the culprit: we'd added an extra validation la...

When Your Docker Image Bloats from 900MB to... Wait, How Did We Get Here?
DevOps & Cloud Aug 2

When Your Docker Image Bloats from 900MB to... Wait, How Did We Get Here?

I was debugging a deployment issue at 2 AM last week when I realized our production Docker image had somehow ballooned to nearly a gigabyte. A gigabyte. For a Node.js API that should've been maybe 150MB lean. That's when I came across this article about image size optimization, a...

Why I'm Finally Committing to Learning in Public (And Why You Should Too)
DevOps & Cloud Aug 1

Why I'm Finally Committing to Learning in Public (And Why You Should Too)

Last month, I spent three hours debugging a CloudFormation template that kept failing in a specific way. The error message was cryptic, the AWS docs were buried under twelve tabs of StackOverflow threads, and I was frustrated. When I finally figured it out—a simple missing proper...