Cascading Failures: How Distributed Systems Break and How to Stop Them

A single slow database query shouldn't take down your entire API. But it does. The payment service times out waiting for the database, retries pile up, the connection pool fills, then the cart service can't reach payments, then the frontend starts hammering the cart service, then your CDN starts seeing errors and caches them. Fifteen minutes later, you have a P1 incident that started with one slow query.

That's a cascading failure. It's one of the most frustrating kinds of outage to deal with because by the time you're paged, the original cause is buried under five layers of symptoms.

Why cascades are different

In a monolith, failures tend to be local. A bad database call crashes the thread; other threads keep running. Recovery is usually straightforward.

Distributed systems don't work that way. Every network call is a potential failure point, and those failures propagate in ways that aren't obvious from looking at any single service. The blast radius of a small problem can expand rapidly, especially under load.

The tricky part: systems often appear healthy right up until they don't. CPU looks fine. Error rates are low. Then a dependency degrades slightly, queues start backing up, and within minutes you're looking at cascading 500s across half your infrastructure.

The patterns that cause cascades

Most cascading failures follow one of a few recognizable patterns. Knowing them helps you spot them faster during an incident.

Retry storms

A service calls a dependency that's slow. The client times out and retries. The dependency is still slow, so it gets even more requests now, which makes it slower, which causes more timeouts, which causes more retries. You've created a positive feedback loop.

Retries are genuinely useful, but without exponential backoff and jitter, they amplify the exact problem they're trying to work around. Uncoordinated retries across thousands of clients can turn a 20% degradation into a full outage.

Thundering herd

Similar to retry storms, but triggered differently. A cache expires or a server restarts, and every client that was waiting suddenly sends requests at the same time. The backend gets a massive spike it wasn't built to handle, buckles, and now everything downstream is waiting too.

This is especially common during incident recovery. You fix the problem, services come back up, and then get slammed by all the backed-up work.

Resource exhaustion

One slow dependency holds connections open. The connection pool fills up. Now services that don't even use that dependency start failing because they can't get database connections for their own queries. The failure has spread to unrelated parts of the system.

Thread pools, file descriptors, memory, and CPU are all finite. When a dependency gets slow instead of fast-failing, it doesn't just hurt the callers, it eats resources that everything else needs.

Latency amplification

In synchronous call chains, latency compounds. If service A calls B which calls C which calls D, a 200ms slowdown in D becomes a 200ms+ slowdown in C, B, and A. Add retries and the problem gets worse. Add high request volume and you've exhausted request queues throughout the chain.

The deeper the call graph, the more exposure you have to this.

Recognizing a cascade during an incident

When you're responding to an incident, a few signals suggest you're dealing with a cascade rather than a single point of failure.

Multiple services failing at the same time. If your alerts fire across several unrelated-looking services simultaneously, the errors are probably symptoms, not causes.

Error rates climbing steadily instead of spiking. A cascade often has a slow start. One service degrades, then another, then another. The timeline has a shape to it.

Upstream services look fine. The service that started the problem might look mostly healthy by the time you're looking at it. Check latency histograms, not just error rates. A p99 that's 10x normal but a p50 that's fine can hide a lot of damage.

Retry metrics are elevated. High retry rates anywhere in the system are a strong signal that something upstream is struggling to respond.

Stopping a cascade in progress

Once you've identified a cascade, the priority is stopping the spread, then fixing the cause.

Drop load, don't retry. If retries are amplifying the problem, turn them off or slow them down. Sometimes the fastest path to recovery is reducing the load hitting the degraded service, even if that means accepting errors temporarily.

Fail fast. If a dependency is clearly down, make callers fail fast instead of waiting. This frees up resources and prevents the cascade from spreading further. Many services have circuit breakers for this; if yours do, check whether they've opened.

Shed load deliberately. Some systems support load shedding, returning errors to low-priority traffic so high-priority requests can get through. If you have this capability, use it. It's better to degrade gracefully than to collapse entirely.

Restart carefully. When you restart a service during a cascade, it can trigger a thundering herd as it reconnects and processes backed-up work. Bring it back up slowly if possible, or have a plan for the spike.

Design choices that limit blast radius

The best time to deal with cascading failures is before they happen.

Set aggressive timeouts everywhere

Default timeouts in most HTTP clients are way too long. A 30-second timeout means a slow dependency can hold a connection for 30 seconds before failing, which is plenty of time for resource pools to fill up. Set timeouts based on what your SLO actually requires, not on what the client library defaults to.

Every external call should have a timeout. Every one.

Use circuit breakers on critical dependencies

A circuit breaker watches for failures and, after a threshold is crossed, stops sending requests to the failing dependency entirely, returning errors immediately instead. This fast-fail behavior prevents resource exhaustion and gives the dependency time to recover.

Most service mesh and RPC frameworks have circuit breaker support built in. If you're not using them, start with your highest-traffic external dependencies.

Add jitter to retries

If you use retries, use exponential backoff with random jitter. This spreads retries out over time instead of creating synchronized waves. The algorithm is simple:

def backoff_with_jitter(attempt: int, base_delay: float = 0.1, max_delay: float = 30.0) -> float:
    delay = min(base_delay * (2 ** attempt), max_delay)
    return delay * (0.5 + random.random() * 0.5)

Isolate failure domains

Use bulkheads to limit the blast radius of a single dependency failure. If service A calls both B and C, give them separate thread pools or connection pools. A slowdown in B shouldn't exhaust the resources that A uses to talk to C.

This is extra infrastructure complexity, so prioritize it for your highest-risk dependencies.

Make your dependency graph explicit

You can't protect against failures in dependencies you don't know you have. A service catalog that tracks upstream and downstream dependencies helps you reason about blast radius during incident triage, and helps you know which circuit breakers and timeouts to prioritize.

If you're seeing a cascade and don't know which service is the root cause, a dependency graph cuts the investigation time considerably.

The recovery phase matters too

After you fix the root cause, watch the metrics carefully as traffic returns to normal. Thundering herds during recovery are common, and a service that just recovered can get knocked back down by the flood of backed-up work.

Rate limit or warm up slowly where you can. If you're restarting a cache, populate it before sending traffic. If a worker queue has backed up, drain it gradually.

Recovery is part of the incident, and how you handle it affects your actual MTTR just as much as how fast you find the root cause.

---

Cascading failures are hard to predict and harder to debug in the moment. The teams that handle them well tend to be the ones that have thought through failure modes in advance: they know their dependency graph, they have circuit breakers configured, and they know how to shed load when things go sideways.

The architecture work takes time. The payoff is incidents that stay contained.