In a distributed system, a failed request does not always mean the operation is impossible. A dependency may be restarting, a connection may have timed out, or a service may be overloaded for a few seconds.
That makes retrying useful. It also makes retrying one of the easiest resilience mechanisms to misuse.
What is retry with backoff?
A retry repeats an operation after a failure. Backoff adds a delay between attempts instead of sending them immediately one after another.
The purpose is simple: give the dependency time to recover and avoid turning one failed request into many failed requests in the same second.
What is exponential backoff?
With exponential backoff, the delay grows after each unsuccessful attempt. The first retry may happen quickly, while later attempts wait progressively longer.
The exact numbers are less important than the behavior: repeated failure should reduce retry pressure rather than maintain a constant flood.
A maximum delay and a maximum number of attempts are usually needed as well. Otherwise a request can keep retrying long after it has stopped being useful to the business.
Why add jitter?
If thousands of clients fail at the same moment and all use the same backoff schedule, they may retry at the same moments too.
Jitter adds randomness to the delay so clients spread their retries over time. The goal is not randomness for its own sake. It is to prevent synchronized retry waves.
Which errors should be retried?
I would retry only when there is a reasonable expectation that time can change the outcome.
Temporary network failures, short overload, or an unavailable dependency can fit that model. Invalid input, missing permission, or a business rule failure usually will not become successful simply because the same request is sent five more times.
Retrying a permanent error adds load, increases latency, and hides the real problem.
Idempotency decides whether retry is safe
Before automatically retrying an operation, ask what happens if the first attempt actually succeeded but the response was lost.
A repeated “read this data” is usually harmless. A repeated “charge this customer” can be very different.
Important write operations often need an idempotency mechanism or another business rule that ensures the same logical request cannot create duplicate side effects.
This is also why retry strategy belongs to the business flow, not only to a generic HTTP client configuration.
What is a retry storm?
A service becomes overloaded and starts failing. Clients retry. The retries add more traffic. The service has even less capacity to recover, so more requests fail and more clients retry.
At that point the resilience mechanism has become part of the incident.
Backoff and jitter reduce this risk, but architecture should also limit how much retry traffic one original request can create.
Retries need a time budget
A user waiting for a page has a different time budget from a background reconciliation job.
If the customer-facing request has a five-second usefulness window, six retries over a minute do not improve reliability. They turn a clear failure into a slow failure.
Retries, timeouts, and the overall business deadline should be designed together.
Retries across several layers can multiply
A browser may retry an API call. The API gateway may retry the service. The service may retry a database or another API. A queue consumer may retry the whole operation again.
Each layer can look reasonable in isolation while the combined behavior creates a large amplification factor during an incident.
Teams should know where retries are allowed and avoid invisible retry chains.
What does the business gain?
Good retry behavior converts short technical failures into successful business operations without involving the customer or support team.
It also helps systems recover more gracefully because temporary trouble does not immediately become a permanent customer error.
The business value is resilience to transient failure, not simply “more attempts.”
What do we pay for it?
Retries consume capacity, increase latency, complicate observability, and can duplicate side effects.
Every retry policy needs ownership: which errors qualify, how long the business can wait, how many attempts are allowed, and how the system behaves after the final failure.
When should you not retry?
- the error is clearly permanent;
- the operation is unsafe to repeat and has no idempotency protection;
- the business deadline has already passed;
- the dependency explicitly tells the client not to retry yet or to slow down;
- another layer already owns the retry policy;
- retrying creates more risk than returning a clear failure.
Questions to ask before implementing retry
- Which exact failures are transient?
- Is the operation safe to repeat if the first attempt actually succeeded?
- What is the total time budget?
- How many attempts are allowed?
- Do we use backoff and jitter?
- Are other layers retrying the same operation too?
- How do we observe retry volume during an incident?
- What happens after the final attempt fails?
In the end
Retry is useful because short failures are normal in distributed systems. It is dangerous because repeated traffic is not free.
A good retry policy gives a dependency time to recover and protects the business from a transient failure. A bad one makes the failing dependency work harder exactly when it needs less load.