The Most Dangerous Assumption in Distributed Systems

One assumption causes more production issues than most engineers realize. "This request will only happen once."

In development, that often feels true. In production, it rarely is.

• Users click twice.
• Browsers retry requests.
• Mobile networks reconnect.
• Load balancers retry failed connections.
• Services resend messages after temporary failures.

Suddenly, the same operation reaches your system more than once.

If your architecture isn't prepared for that, strange things start happening.

• Duplicate orders
• Double payments
• Multiple emails
• Repeated notifications
• Inconsistent data

One lesson that's changed how I think about distributed systems is this: Every request should be treated as if it might arrive again.

That doesn't mean every operation must be idempotent. Reading a product page is very different from charging a credit card.

But whenever an action changes state, it's worth asking: "What happens if this exact request is processed twice?"

That single question has prevented countless production issues.

Good architecture isn't just about making the first request succeed. It's about making repeated requests safe.

That's one of the biggest differences between software that works in testing and software that survives real-world traffic.

What's one production issue you've seen that was caused by an assumption that turned out not to be true?