The Network Is Not Reliable. Design Like It

When engineers first design a system, there's usually one assumption hiding in the background:

• Everything will work.
• The database will respond.
• The API will be available.
• The network will be stable.
• The downstream service will return data. Until one day it doesn't.

And that's when system design becomes real. Over the years, I've noticed that most production incidents aren't caused by complex algorithms or obscure bugs. They're caused by assumptions.

• Assumptions that a dependency would always be available.
• Assumptions that a request would always succeed.
• Assumptions that a timeout would never happen.

In distributed systems, failures aren't edge cases. They're expected behavior. That's why resilient systems use patterns like:

• Timeouts
• Retries
• Circuit breakers
• Fallbacks

Not because failure is likely. Because failure is inevitable.

One idea changed how I think about architecture: A good system isn't one that never fails. It's one that fails gracefully.

Users may never notice a successful request. But they'll always remember an unavailable system. And that's why reliability is ultimately a design decision.

If you've followed this series, you've already learned more system design than most engineers realize.

Question: What's the biggest lesson you've learned from a production failure?