I Learned the Hard Way: Why "Pending" Saved My Career (and My Client's Money)

Two years ago, I shipped a payout system that I was genuinely proud of. It was fast, it was clean, and it handled payments with what I thought was bulletproof logic. A user requests money. We hit the bank API. Success or failure. Move on. Then one Tuesday morning at 3 AM, my phone started buzzing. A client had discovered we'd paid out the same amount twice to some of their contractors. My stomach dropped. I hadn't made a logic error—I'd made something worse. I'd made an architectural assumption that cost real money.

That incident forced me to completely rethink how I design payment flows. I spent the next month rewiring our entire system around a principle I'd honestly overlooked: uncertainty is not a failure. It's a state you need to plan for.

The Synchronous Illusion That Kills Payment Systems

Most of us spend our careers building systems that are synchronous by nature. Request comes in. Response goes out. That's the mental model we're trained on. Payment infrastructure deliberately doesn't work that way, and if you treat it like it does, you're going to have a bad time.

When I first integrated with our payment gateway, I expected each API call to be definitive. Bank says success? Great, we're done. Bank times out? Obviously failed. But what actually happened was more complex. That timeout might have occurred after the bank already processed the payout. The money is gone. My system thinks it never left. Cue the duplicate payment problem.

The reality is that money moves through multiple systems—settlement layers, batch processors, fraud checks, confirmations from the recipient's bank. A response from the first integration point tells you almost nothing about the final state. It tells you whether that specific layer responded, not whether the money actually landed.

The Third State Nobody Plans For

This is where most teams trip up. We think in binary: success or failure. Payment systems need a third option: unknown.

Unknown is uncomfortable for engineers. It feels like technical debt. It feels unresolved. But unknown is actually the honest state of many transactions for hours after they're initiated. The payout might succeed. It might fail. It might still be processing somewhere in the chain.

I now explicitly model this in every system I build. Instead of forcing transactions into success/failure as quickly as possible, I let them live in a PENDING state while I wait for confirmation from upstream sources. This feels slower, but it's actually the fastest path to correctness.

Why False Negatives Are More Expensive Than Delays

Here's the asymmetry that changed how I think about payment systems: treating a successful transaction as failed is exponentially worse than treating an uncertain transaction as pending.

If I mark something as failed that actually succeeded, I've now got duplicate money in flight. Recovery is a nightmare. We're talking manual investigations, coordination with banks, potential regulatory reporting issues, and customer refunds. If instead I just... wait for confirmation, the worst case is that a user's transaction takes longer. That's annoying. It's not a financial disaster.

I made the decision years ago that my payout systems would optimize for financial correctness over perceived speed. Users can be patient. Banks can be corrected. Money out the door twice cannot be unsent.

Building Reconciliation Into Your DNA

The most important architectural shift I made was treating reconciliation not as a cleanup phase but as a core system component. Every payout I send, I'm continuously checking: did this actually land? Do my records match what the bank says? Are there discrepancies I need to investigate?

Here's a simplified version of how I structure this now:

// Payout moves through explicit states
const PAYOUT_STATES = {
  CREATED: 'created',
  PROCESSING: 'processing',
  PENDING_CONFIRMATION: 'pending_confirmation',
  SUCCESS: 'success',
  FAILED: 'failed'
};

// Instead of relying solely on the initial API response,
// we continuously reconcile against upstream sources
async function reconcilePayoutStatus(payoutId) {
  const local = await getLocalPayout(payoutId);
  const upstream = await bankAPI.getStatus(payoutId);
  
  // Only transition out of PENDING if we have definitive proof
  if (upstream.status === 'SETTLED') {
    await updatePayout(payoutId, PAYOUT_STATES.SUCCESS);
  } else if (upstream.status === 'REJECTED') {
    await updatePayout(payoutId, PAYOUT_STATES.FAILED);
  }
  // Otherwise: stay in PENDING_CONFIRMATION
}

The key here is that I'm not trusting my initial response. I'm treating reconciliation as an ongoing process that continuously checks the source of truth.

My Take

I went from thinking reconciliation was something you did at the end of the day to understanding it's the foundation of payment architecture. The systems I'm most confident in now are the ones that are most comfortable with uncertainty—the ones that can say "I genuinely don't know yet" and handle that gracefully.

The original article's core principle resonates deeply with production experience: conservative is faster when you measure in "how many incidents did we have" instead of "how many milliseconds did transactions take."

What's your approach to handling uncertain states in payment systems? Do you have battle scars from shipping something that assumed synchronous behavior?

Source: This post was inspired by "The Golden Rule of Payout Systems: Why "Pending" is Never a Failure" by Dev.to. Read the original article