An integration retries document submissions to a supplier. When a request fails, the application tries again. With occasional faults, this is convenient: employees never notice the interruption. Then the supplier starts responding slowly, and almost every original request generates additional attempts. Recovery traffic becomes a substantial part of the load.
Retries have two responsibilities: improve the chance of completing one operation and avoid obstructing recovery of the service being called. A “try several times” setting only solves the first under certain conditions. It can make the second much harder, even when every request is legitimate.
What might change before the next attempt?
A temporarily unavailable connection may work on the next attempt. A document rejected for a missing required field will still be invalid after a delay. An access denial normally requires a change in credentials or permissions, not endless repetition of the same request.
A generic server-error label is not enough to classify every failure. The specific API contract needs to identify temporary rejections, final failures, operation-status queries and overload behaviour. A supplier may indicate when another request is appropriate, for example through Retry-After. Ignoring that advice to follow a shorter local schedule defeats the purpose.
The retry must also remain useful to the business. Once a customer has received a final rejection, submitting the document silently later may be as surprising as losing it. Background workflows can allow longer recovery periods, but they still need visible state and a stopping policy.
The response disappeared. The document may already exist
A timeout does not tell the caller which side failed. The supplier may have registered the document and lost the acknowledgement. Resubmitting it as a new operation can create another document.
Such actions need a stable identifier of intent and an agreed response to repeated attempts. The same key belongs to retries of the same operation; generating a new key after every network error defeats that protection. When the outcome is unknown, querying its status may be safer than sending the command again immediately.
This is the job of idempotency. A delay is not a substitute. Perfectly spaced requests can still repeat a business effect if the recipient cannot recognise them. For an external API, check the key's scope, retention period and treatment of changed parameters, not merely whether an idempotency header exists.
Identical delays can synchronise the next wave
Backoff introduces a pause after failure. Exponential backoff increases the delay with successive failures, usually up to a cap. That gives the dependency room to recover and reduces one caller's request frequency.
But many application instances can fail at the same instant and count down identical pauses. The next wave then arrives together too. Jitter introduces random variation into the schedule so attempts spread over time. There are several algorithms; the relevant test is whether requests actually remain dispersed, including after the maximum delay is reached.
Neither backoff nor jitter reduces the number of customers needing service. If original demand remains above the supplier's capacity, spacing retries will not be sufficient. Additional retry traffic and accepted work may need their own limits, with some operations deferred or rejected.
Which other layers are already retrying?
A client library, gateway, application service and queue consumer can each have a retry policy. Every policy may seem reasonable alone. Together, they can multiply calls to the final dependency.
For illustration, three nested layers allowing three total attempts each, including the first call, can generate up to 27 downstream calls for one top-level request. That is a worst-case calculation when repeatable failure persists across all layers, not the normal cost of every request. It explains why an audit must include library defaults rather than only application code.
Decide which layer owns retries of the business operation and constrain independent attempts elsewhere. All calls and pauses need to fit inside the overall operation deadline. Starting another attempt when there is no longer enough time to use a plausible successful response achieves little.
A per-operation limit is not a global safety limit
A modest number of attempts can still produce a large retry stream when many callers fail together. An aggregate retry budget limits how much additional work an application is prepared to send to a dependency over a period. Once it is exhausted, some requests finish without another attempt, preserving capacity for fresh work and recovery.
Long-running background retries should not hold a database connection or transaction during the waiting period. Persist the job state and schedule another attempt separately. That introduces responsibilities of its own: job expiry, duplicate protection and a path for work that cannot complete. Putting something in a queue does not authorise unlimited retries.
If a dependency stays unhealthy, a circuit breaker may temporarily stop normal calls. The retry loop should respect that result. Otherwise one part of the system is trying to protect the dependency while another keeps attempting to get around the protection.
Measure recovered work, not activity
Track original operations, additional attempts and final successful outcomes separately. A system that hides a rising first-attempt failure rate behind retries can look healthy until it reaches a much more serious limit. Also measure end-to-end duration, duplicate effects and the age of unfinished background work.
Test a brief interruption, prolonged unavailability and a lost acknowledgement after successful document acceptance. Then restore the dependency and check that the backlog does not arrive as one sudden burst. The recovery phase is where the retry policy demonstrates whether it actually helps the service return to normal.