RELIABILITY · DISTRIBUTED SYSTEMS · AVAILABILITY

Retries do not repair an outage

KEY TERMS transient faults / retry budgets / exponential backoff / jitter / circuit breakers / load shedding

A retry feels responsible. Something failed, so try again. Maybe the network blinked. Maybe the next server is healthy. Maybe the response was already on its way when the timeout fired.

Sometimes that instinct is right. The dangerous part is treating it as universally right.

A retry does not repair an outage. It sends another request into the conditions that produced the first failure. If those conditions were brief and isolated, that may be enough. If the dependency is overloaded, each retry is extra work for the component already struggling to keep up.

A failure is not a diagnosis

“The request failed” is not enough information to choose a retry policy. A timeout might mean a transient network problem. It might also mean a server is so overloaded that it started work too late to finish before the caller gave up. A 503 might describe a brief deployment gap, a saturated pool, or a dependency that will be unavailable for hours.

Retries have a chance only when the underlying condition is likely to change before the next attempt. Microsoft’s transient-fault guidance makes this distinction directly: retry policies are for faults expected to clear by themselves, while persistent failures need a different response. Transient fault handling.

That sounds obvious, but a client rarely receives a neat label saying “this will recover in 80 milliseconds.” A useful retry policy therefore starts by being conservative about what it calls transient.

The multiplier nobody sees in the happy path

One retry is not always one extra request.

Imagine a browser calls an API, the API calls a service, and the service calls a database. If every layer retries three times, one user action can turn into dozens of database attempts. The layers do not know that the layers below them are already retrying. Each one is merely following a reasonable local rule.

That is how an error becomes traffic.

Google’s SRE guidance describes the same failure pattern: retries can keep an overloaded backend overloaded, consume resources on work that will fail anyway, and grow into a cascading failure. It recommends limiting retries per request, using a retry budget, and avoiding retries at multiple layers of a stack. Cascading failures.

The lesson is not “never retry.” It is that retry is a load-generating feature. It deserves the same capacity thinking as a new endpoint.

Backoff changes the shape of the damage

Immediate retries are the most tempting and usually the worst default. A short network blip can make thousands of clients fail at roughly the same moment. If all of them retry on the same fixed schedule, they return together and make the recovery spike sharper than the original problem.

Exponential backoff spreads attempts farther apart. Jitter adds randomness so clients do not line up again on the next interval. Neither makes an unavailable service healthy. They reduce the chance that the recovery itself is crushed by a synchronized crowd of callers.

There is a second limit that matters just as much: stop. A request should have a bounded attempt budget, and the process should have a broader retry budget. If a meaningful share of a client’s traffic is already retries, the problem is probably not one unlucky request anymore. Google documents both per-request and per-client retry budgets as a way to contain retry growth under overload. Handling overload errors.

Retrying at the right boundary

The layer closest to a dependency is usually the one with the best information about that dependency’s failure. It can see whether a failure was a connection reset, a timeout, a rate-limit response, or a permanent validation error. Higher layers usually see only “something below failed.”

That is why retries should have one owner. If a database call is retried, the service directly above the database should own that policy. Once it has exhausted its budget, it should return a response that tells the caller not to repeat the same work blindly. A higher layer may still choose a fallback or a degraded response, but it should not silently restart the same retry ladder.

This also means that retries need observability. Count attempts separately from user requests. Record why an attempt was retried. Track the fraction of traffic that is retry traffic. Without that, a graph can make an outage look like rising demand when it is actually the system calling itself more often.

When failure is the safer answer

There is a point where returning an error is kinder to the system than trying harder. A fast 429, 503, or degraded response can release threads, connections, and queues. A request held open while several layers wait and retry occupies resources that may be needed by work that still has a chance to succeed.

Circuit breakers, concurrency limits, and load shedding are not admissions of defeat. They are ways to stop a local failure from becoming shared exhaustion. The goal is not to produce a success response at any cost. The goal is to preserve enough capacity that recovery remains possible.

A retry policy worth explaining

A policy should be simple enough that its costs are visible:

• Retry only failures that are plausibly transient.
• Keep the attempt count bounded and explicit.
• Use exponential backoff with jitter.
• Give retries a budget at the process or client level.
• Retry at one layer, close to the failing dependency.
• Respect overload and rate-limit signals instead of treating them as invitations to push harder.
• Measure retry traffic separately from original traffic.

The useful question is not “did we add retries?” It is: under which failure does another attempt have a real chance of helping, and who pays if it does not?

Related implementation

R02: API resilience under dependency failure is a local study of bounded retry, timeout, circuit-breaker, and concurrency-limit behaviour. Its recorded results are separate from the sources and argument in this article.