A retry policy is the small rulebook a program follows when something fails temporarily: try again, but only in the right way, only so many times, and only when it is safe to do so.

Imagine you call a friend and the line is busy. You do not give up on the friendship forever — you hang up and call again a minute later. That small, everyday decision, “try again after a short wait,” is the entire idea behind a retry policy. In software, a retry policy is a set of rules that tells a program exactly how to behave when an operation — a network call, a database write, a file read — fails temporarily. It answers three questions: should we try again at all, how long should we wait before trying, and when should we finally give up?

The idea is almost as old as computer networking itself. Early packet-switched networks in the 1970s, including the ARPANET, had to deal with lost or corrupted packets over unreliable links. Engineers built retransmission logic directly into protocols like TCP: if an acknowledgment did not arrive in time, the sender would resend the packet. This was, in effect, the first widely deployed retry policy, and many of its ideas — timeouts, exponential backoff, giving up after too many attempts — are still the backbone of retry logic today.

As software moved from single machines to networks, and later from monoliths to microservices and cloud-native systems, the number of things that could fail temporarily exploded. A single user action might now trigger dozens of network calls across services, databases, caches, and third-party APIs. Each of those calls can fail for reasons that have nothing to do with a real, permanent problem: a server was briefly overloaded, a network packet got dropped, a load balancer was mid-rotation. Retry policies became a standard, almost mandatory, part of building dependable distributed systems.

It is worth pausing on why this rulebook needs to exist at all, rather than each engineer just writing a loop that says “try three times” whenever they feel like it. The answer is consistency and safety. Distributed systems are made of hundreds or thousands of individual call sites — a service calling a database, a service calling another service, a mobile app calling an API. If every one of those call sites invents its own ad-hoc retry behaviour, the system as a whole becomes unpredictable: some parts retry too aggressively and overload dependencies, others do not retry at all and surface avoidable errors to users, and nobody can reason about the worst-case load a dependency might see during a bad day. A retry policy, especially when implemented as a shared, well-tested library or a piece of shared infrastructure like a service mesh, turns that chaos into something consistent, tunable, and observable.

The concept also travelled well beyond raw networking. Database drivers retry on deadlocks. Message queue clients retry on broker unavailability. Mobile apps retry on flaky cellular connections. Command-line tools like curl and package managers like npm or Maven retry downloads that fail partway through. Wherever a program depends on something outside its own process — and almost every modern program does — the same basic questions from the introduction above show up again: should this failure be retried, how long should we wait, and when do we stop?

Why this topic deserves careful study

It is tempting to treat retries as a solved, boring problem — “just wrap it in a loop.” In practice, retry policy design sits at the intersection of several genuinely hard problems: distributed systems theory (what does “success” even mean when a request might have partially succeeded?), queueing theory (how does added retry traffic affect a system already near its capacity limit?), and human factors (how do you make a hidden mechanism visible enough that engineers trust it during an incident, but invisible enough that it does not clutter normal operation?). This guide walks through all of these angles, from the everyday building blocks to the subtle failure modes that only show up at scale.

Why can a program not simply fail immediately and let a human deal with it? Because most failures in a distributed system are not permanent — they are transient, and a well-designed retry absorbs them before anyone notices.

Why can a program not just fail immediately and let the human deal with it? Because most failures in a distributed system are not permanent — they are transient. A transient failure is one that resolves itself shortly after it happens: a server was momentarily too busy, a network link had a brief hiccup, a database connection pool was exhausted for half a second. If a program gives up on the very first sign of trouble, it turns tiny, self-healing blips into visible, user-facing errors.

Consider an online store’s checkout flow. The payment service might reject a request for a fraction of a second because it is restarting one of its ten instances during a routine deployment. Without a retry policy, every customer whose request happened to land on that instance during that window sees “Payment failed.” With even a simple retry policy, the request quietly succeeds a moment later on a healthy instance, and the customer never notices anything happened.

What kinds of failures are we talking about?

• Network-level failures — dropped packets, DNS hiccups, connection resets, timeouts.
• Server overload — the target service is temporarily out of capacity and returns a “too busy” response.
• Infrastructure churn — a container is being restarted, a pod is being rescheduled, a load balancer is updating its routing table.
• Rate limiting — the caller has been asked to slow down and try again later.
• Deadlocks and contention — a database transaction was rolled back because it collided with another transaction.

Not every failure belongs on this list. If a request is rejected because the input was invalid, the user is not authorised, or the resource genuinely does not exist, retrying will never help — it will just waste time and resources while producing the exact same failure again. Knowing the difference between a failure worth retrying and one that is not is, as we will see, the single most important design decision in a retry policy.

The cost of not retrying

It helps to think through the alternative. Suppose a system has no retry logic anywhere. Every transient blip — and in a system with hundreds of moving parts, transient blips happen constantly, just from routine deployments, garbage collection pauses, network reconfiguration, and normal load fluctuation — becomes a visible failure. Support tickets pile up for “random” errors that, on investigation, turn out to correlate with deployment windows or brief load spikes. Engineers get paged for problems that would have resolved themselves within a second. Over time, teams either build ad-hoc, inconsistent retry logic scattered across the codebase (the worst outcome, since nobody can reason about it holistically), or they under-invest in reliability altogether because “the system is just flaky” becomes an accepted, if false, belief.

Framing the goal precisely

The motivation for a retry policy, stated precisely, is this: absorb the failures that are statistically likely to resolve themselves within a short window, without meaningfully increasing the risk or cost of the failures that will not. Every design decision in the rest of this guide — how long to wait, how many times to try, which errors qualify — is really just a way of drawing that line as accurately as possible for a given system’s actual failure characteristics.

Each of these terms shows up constantly in retry policy design; understanding them well makes everything after this chapter much easier to follow.

Idempotency

An operation is idempotent if doing it once has the same effect as doing it many times. Pressing an elevator’s “up” button repeatedly does not call five elevators — the effect of pressing it once and pressing it five times is identical. In software, an HTTP GET request (just reading data) is naturally idempotent. But a POST that creates a new order is not: if you retry it blindly, you might charge the customer twice and create two orders. Idempotency is the single biggest prerequisite for safe retries.

Backoff

Backoff is the strategy for how long to wait between attempts. The simplest form, fixed backoff, waits the same amount of time every time (e.g., always 500ms). A far more common and safer form is exponential backoff, where the wait time roughly doubles after each failed attempt (e.g., 200ms, 400ms, 800ms, 1600ms). This gives an overloaded service progressively more breathing room to recover.

Jitter

If a thousand clients all fail at the same moment and all retry using the exact same backoff schedule, they will all hammer the server again at the exact same moment — and likely fail again, together, forever. This synchronised pile-up is called a retry storm or the “thundering herd” problem. Jitter fixes this by adding a small amount of randomness to each wait time, so retries from different clients get spread out instead of arriving in lockstep.

Maximum attempts (retry budget)

A retry policy must eventually give up. The maximum attempts setting caps how many times an operation will be retried before the caller is told it failed for good. A related, more advanced idea is a retry budget: rather than letting every single request retry independently, the system tracks the overall ratio of retries to original requests and stops issuing new retries once that ratio gets too high — protecting the whole system, not just one call.

Retryable vs. non-retryable errors

A good retry policy classifies errors before deciding to retry. Errors like “service temporarily unavailable” (HTTP 503) or “request timed out” are usually retryable. Errors like “bad request” (HTTP 400) or “not authorised” (HTTP 401) are not — retrying them will not change the outcome.

Circuit breaker

A circuit breaker is a companion pattern that works alongside retries. It watches the failure rate of calls to a dependency, and if failures cross a threshold, it “trips” and stops sending requests entirely for a while, failing fast instead. This prevents retries from continuing to pound a dependency that is clearly down, and gives the dependency room to recover.

Deadline / overall timeout

Separate from the per-attempt timeout, a deadline is a ceiling on the total time an entire retry sequence is allowed to take, across all attempts combined. Without a deadline, a policy with a generous max-attempts count and a growing backoff could theoretically take tens of seconds or more to finally give up — long after the calling user or upstream system has stopped waiting. A deadline guarantees a predictable worst-case latency for the whole operation, not just for one attempt.

Jittered exponential backoff, precisely

Because this specific strategy is so common, it is worth spelling out the formula plainly. Given a base delay b, a multiplier m (usually 2), and an attempt number n, the raw exponential delay is b × m^(n-1). Full jitter, one of the most recommended variants, then picks a wait time uniformly at random between 0 and that raw value, rather than just adding a small random offset on top. This maximises the spreading effect between competing clients and has been shown, in practice, to reduce contention more effectively than smaller jitter ranges.

Retry-After headers

Some servers do not just say “come back later” — they say exactly how long to wait. An HTTP 429 Too Many Requests or 503 Service Unavailable response can include a Retry-After header specifying a number of seconds or an exact timestamp. A well-behaved retry policy checks for this header and honours it instead of using its own computed backoff, because the server is in the best position to know when it will actually be ready again.

A retry policy is not a single line of code — it is a small system with several interacting pieces. Understanding each piece separately makes it much easier to configure or debug a retry policy in a real application.

The components, one by one

• Trigger / error classifier — inspects the failure (exception type, HTTP status code, error code) and decides whether it qualifies for a retry.
• Backoff strategy — a function that, given the attempt number, returns how long to wait: fixed, linear, or exponential, usually with jitter mixed in.
• Attempt counter / retry budget — tracks how many attempts have been made for this call, and enforces the maximum.
• Timeout — each individual attempt should have its own timeout so a single hung attempt cannot stall the whole retry loop forever.
• Circuit breaker (optional but common) — sits alongside the retry logic and can short-circuit new attempts entirely if the dependency is clearly unhealthy.
• Observability hooks — logging and metrics that record every attempt, every failure reason, and every final outcome.

A typical error classification table

Because the error classifier is the component that decides whether any of the rest of this architecture even runs, it is worth seeing a concrete example of what its rules usually look like for an HTTP-based service.

Notice that even within a single status code, judgment is sometimes required — a 500 caused by a genuine, deterministic application bug will simply fail the same way on every retry, wasting attempts. Some sophisticated policies track the historical retry-success-rate per error type and adjust their classification rules over time based on real data rather than a fixed, hand-written table.

Let us walk through exactly what happens, step by step, when a retry-wrapped call is made. Suppose a service calls another service’s API to fetch a user’s profile.

If attempt 2 had also failed with a retryable error, the loop would repeat: classify, check budget, back off longer, retry — until either a success happens or the maximum attempts are exhausted, at which point the original (or a wrapped) error is finally surfaced to the caller.

A minimal retry loop in Java

public class RetryExecutor {

    public static <T> T executeWithRetry(Callable<T> operation,
                                          int maxAttempts,
                                          long baseDelayMillis) throws Exception {
        int attempt = 0;
        while (true) {
            attempt++;
            try {
                return operation.call();
            } catch (Exception e) {
                if (!isRetryable(e) || attempt >= maxAttempts) {
                    throw e; // give up: non-retryable or out of attempts
                }
                long backoff = computeBackoffWithJitter(attempt, baseDelayMillis);
                Thread.sleep(backoff);
            }
        }
    }

    private static boolean isRetryable(Exception e) {
        return e instanceof java.net.SocketTimeoutException
            || e instanceof java.io.IOException;
    }

    private static long computeBackoffWithJitter(int attempt, long baseDelayMillis) {
        long exponential = baseDelayMillis * (long) Math.pow(2, attempt - 1);
        long jitter = (long) (Math.random() * exponential * 0.3);
        return exponential + jitter;
    }
}

This tiny class captures the essence of nearly every production retry library: classify the error, check the budget, compute a backoff with jitter, wait, and try again.

A production-grade example with Resilience4j

Hand-rolled retry loops are fine for learning, but production Java systems almost always reach for a battle-tested library instead, since it is easy to get subtle details wrong (thread blocking, exception wrapping, metric emission). Resilience4j is a popular, lightweight library for exactly this purpose.

RetryConfig config = RetryConfig.custom()
    .maxAttempts(4)
    .intervalFunction(IntervalFunction.ofExponentialRandomBackoff(
        Duration.ofMillis(200), // initial interval
        2.0,                    // multiplier
        0.3))                   // jitter factor
    .retryExceptions(SocketTimeoutException.class, IOException.class)
    .ignoreExceptions(IllegalArgumentException.class)
    .build();

RetryRegistry registry = RetryRegistry.of(config);
Retry retry = registry.retry("inventoryService");

Supplier<InventoryResponse> decorated = Retry.decorateSupplier(
    retry,
    () -> inventoryClient.checkStock(sku)
);

InventoryResponse response = decorated.get();

Note how the configuration mirrors exactly the components discussed in the architecture chapter: a maximum attempt count, an interval function combining exponential growth with jitter, and an explicit list of which exceptions are retryable versus which should be ignored (surfaced immediately). Resilience4j also emits metrics automatically for every retry attempt, which feeds directly into the monitoring practices covered later in this guide.

Zooming out from a single call, it helps to see how a retry policy behaves over the lifetime of a request as it moves through a distributed system with multiple hops.

Notice something important here: only the Order Service → Payment Service hop retries. The client and the gateway do not need to know a retry ever happened. This is a key design principle: retries should generally happen at the layer closest to the failure, not be re-triggered redundantly at every layer above it — otherwise a single failure at the bottom can multiply into an exponential number of retries as it echoes up through the call stack (sometimes called a “retry amplification” problem).

The healthy pattern is to be explicit about which layer owns the retry, and to have every other layer simply propagate the failure upward with a helpful error. In practice, that owner is usually either the client library that talks directly to a resource (a database driver, a payment SDK, a service mesh sidecar) or the outermost edge that receives the user request — almost never both, and almost never everyone in between.

Retries trade a small amount of extra latency and load for a large improvement in reliability — but only when tuned correctly.

The central trade-off is this: retries trade a small amount of extra latency and load for a large improvement in reliability — but only when tuned correctly. Too aggressive, and you risk turning a small hiccup into a full outage. Too conservative, and users see errors that a slightly longer wait would have avoided.

Retries interact with system capacity in ways that are easy to underestimate. Under normal load the overhead is invisible; under a partial outage, it can be the difference between graceful degradation and total collapse.

Every retried request consumes real resources — CPU, network sockets, database connections — a second (or third, or fourth) time. Under normal conditions this overhead is negligible. Under a partial outage, it can be the difference between graceful degradation and a total collapse.

The retry storm

Picture a service that is running at 95% capacity. A brief blip causes 10% of requests to fail. If every failed request is retried once, that adds roughly 10% more load on top of an already-strained service — which can push it past its capacity, causing more failures, which causes more retries, in a vicious feedback loop. This is exactly the same dynamic as a stampede: everyone reacting to a small disturbance ends up causing a much bigger one.

Mitigation techniques

At scale, teams often track a metric called the retry amplification factor: the ratio of total attempts (including retries) to original requests. Keeping this number low and stable, especially during incidents, is one of the clearest signals that a retry policy is well-tuned.

A queueing-theory intuition

There is a useful, simplified way to think about why retries become dangerous specifically near full capacity, rather than at low load. Imagine a service modelled as a simple queue: requests arrive, wait if the server is busy, get processed, and leave. As the arrival rate approaches the service’s maximum processing rate, queueing theory tells us that the average wait time does not grow gradually — it grows explosively, approaching infinity as utilisation approaches 100%. Retries effectively increase the arrival rate. At low utilisation (say, 30%), adding 10% more traffic from retries barely moves the needle. At high utilisation (say, 90%), that same 10% increase can push the system past the point where queueing delay explodes, causing a small nudge to produce a dramatically outsized effect. This is precisely why retry storms tend to appear suddenly, seemingly out of nowhere, once a system is already running hot.

Capacity planning with retries in mind

Because of this dynamic, capacity planning that ignores retry traffic is capacity planning based on a fiction. If a fleet is sized to handle exactly the expected peak request rate with no headroom, it has effectively no room to absorb the extra load that its own retry policy will generate the moment any instability begins — right when that headroom is needed most. A common rule of thumb is to provision enough spare capacity to absorb the maximum plausible retry amplification factor, not just the baseline traffic.

Retry policies are a foundational building block of highly available systems, but they work best as one layer of a broader reliability strategy, not a standalone fix.

How retries combine with other HA techniques

• Redundancy — retries are far more effective when a retried request can land on a different healthy replica instead of hitting the same broken instance again.
• Health checks — load balancers that quickly remove unhealthy instances reduce how often retries are even needed.
• Failover — in multi-region systems, a retry policy might escalate from “retry same region” to “fail over to a backup region” after enough local failures.
• Graceful degradation — when retries are exhausted, a well-designed system falls back to a cached or simplified response rather than a hard error.

A useful mental model: retries handle failures measured in milliseconds to a few seconds. Circuit breakers handle failures measured in seconds to minutes. Failover and redundancy handle failures measured in minutes to hours. Each mechanism is tuned for a different timescale of trouble.

The circuit breaker state machine

Because circuit breakers are such a close partner to retry policies, it is worth understanding their internal states in a bit more detail. A circuit breaker typically has three states. In the closed state, everything is normal: calls pass through, and the breaker just counts successes and failures. If the failure rate crosses a configured threshold, the breaker trips to the open state, where it rejects calls immediately without even attempting them — this is what protects a struggling dependency from any further load, including retries. After a cooldown period, the breaker moves to a half-open state, where it allows a small number of trial calls through to test whether the dependency has recovered. If those succeed, the breaker closes again; if they fail, it reopens and waits longer before trying again.

When a retry policy is combined with a circuit breaker, the breaker’s state should be checked before a retry attempt is made, not just before the very first attempt. Without this check, a retry loop could keep attempting calls against a dependency that the breaker has already correctly identified as unhealthy — defeating the entire purpose of having a breaker in the first place.

Multi-region failover and retries

In systems that span multiple geographic regions for disaster recovery, retry policies sometimes escalate progressively: the first couple of attempts stay within the same region (cheapest, lowest latency), and only after those are exhausted does the policy fall back to retrying against a secondary region. This keeps the common case fast while still providing a safety net for regional-scale outages, without paying the latency and consistency cost of cross-region calls on every single request.

Retry logic is not just a performance concern — it has real security implications that are easy to overlook.

Never blindly retry authentication failures

Retrying a failed login or an expired-token error without any change is not just useless, it can look identical to a brute-force attack to the very system you are calling, potentially triggering account lockouts or security alerts. Authentication failures should typically not be retried automatically at all — or only after explicitly refreshing credentials first.

Replay and duplicate-side-effect risks

Retrying a non-idempotent write — like “charge this credit card” — can cause real financial or data harm if the first attempt actually succeeded but the response was lost before the caller found out. This is why production payment and messaging systems require an idempotency key: a unique identifier attached to the request so the server can recognise “I have already done this exact operation” and return the original result instead of doing it again.

// Attaching an idempotency key so a retried
// payment request can never be double-charged.
HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("https://api.payments.example.com/charge"))
    .header("Idempotency-Key", orderId.toString())
    .POST(HttpRequest.BodyPublishers.ofString(paymentJson))
    .build();

Denial-of-service surface

An attacker who can trigger client-side retries in bulk (for example, by causing a service you depend on to fail intermittently) can effectively weaponise your own retry policy against a downstream system — a self-inflicted denial of service. Retry budgets and circuit breakers act as a safety valve against exactly this scenario.

Interaction with rate limiting

Rate limiting exists to protect a service from being overwhelmed, whether by a legitimate burst of traffic or by abuse. A retry policy that ignores rate-limit responses and keeps retrying at full speed can look, from the server’s point of view, indistinguishable from an attacker probing or attempting to overwhelm the endpoint. This is exactly why honouring a Retry-After header (introduced earlier) matters for security as much as for politeness — it signals that the client is a well-behaved participant in the system rather than something to be blocked or throttled further.

Sensitive data in retry logs

Because good observability practice means logging details about each retry attempt, it is worth remembering that request payloads sometimes contain sensitive data — credentials, personal information, payment details. Retry logging should follow the same redaction and access-control rules as any other logging in the system; a retry wrapper is not exempt from an organisation’s data-handling policies just because its purpose is operational rather than business-facing.

A retry policy that is not observable is a retry policy that will eventually cause a mystery outage — because retries hide failures from the end user by design, they can just as easily hide failures from the engineers who need to know about them.

Key metrics to track

Logging practices

• Log each attempt with a shared correlation / trace ID so all attempts for one logical request can be reconstructed.
• Include the failure reason and the computed backoff duration on each retry log line.
• Emit a distinct event when retries are exhausted, so alerting systems can distinguish “recovered after retry” from “failed for good.”

Instrumenting a Java retry wrapper for metrics

Whether using a library like Resilience4j (which emits metrics out of the box) or a hand-written retry loop, the key is making sure every attempt — not just the final outcome — produces a signal.

public class InstrumentedRetryExecutor {

    private final MeterRegistry meterRegistry; // e.g. Micrometer

    public <T> T executeWithRetry(String operationName,
                                   Callable<T> operation,
                                   int maxAttempts,
                                   long baseDelayMillis) throws Exception {
        int attempt = 0;
        while (true) {
            attempt++;
            try {
                T result = operation.call();
                meterRegistry.counter("retry.attempts",
                    "operation", operationName, "outcome", "success").increment();
                return result;
            } catch (Exception e) {
                meterRegistry.counter("retry.attempts",
                    "operation", operationName, "outcome", "failure").increment();

                if (!isRetryable(e) || attempt >= maxAttempts) {
                    meterRegistry.counter("retry.exhausted",
                        "operation", operationName).increment();
                    throw e;
                }
                long backoff = computeBackoffWithJitter(attempt, baseDelayMillis);
                Thread.sleep(backoff);
            }
        }
    }
    // isRetryable() and computeBackoffWithJitter() as shown earlier
}

With counters like these flowing into a metrics backend, a dashboard can plot retry rate, exhaustion rate, and attempts-per-request over time — turning what would otherwise be an invisible mechanism into one of the most useful early-warning signals an on-call engineer has.

Correlating retries with distributed traces

In a tracing system such as OpenTelemetry, each attempt within a retry sequence can be recorded as its own span, nested under a parent span representing the overall retried operation. This lets an engineer looking at a slow or failed trace see, at a glance, exactly how many attempts were made, how long each one took, and what error each one produced — without needing to cross-reference separate log files by timestamp.

Nearly every major cloud SDK and RPC framework ships with retry policies built in, because transient failures are simply a fact of life at cloud scale.

Configuring retries at the infrastructure layer (like a service mesh) has a big advantage: teams do not need to reimplement retry logic in every service and every language. Envoy, for instance, lets operators define retry policy per route, including which HTTP status codes are retryable and how many retries are allowed, without touching application code at all.

Example: gRPC service config retry policy

{
  "methodConfig": [{
    "name": [{"service": "orders.OrderService"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.2s",
      "maxBackoff": "3s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE", "DEADLINE_EXCEEDED"]
    }
  }]
}

Notice the same core ingredients we have discussed throughout: an initial delay, a multiplier for exponential growth, a ceiling on the maximum backoff, a maximum attempt count, and an explicit list of which errors qualify as retryable.

Retry behaviour interacts closely with the surrounding infrastructure, and getting this interaction wrong is a common source of subtle bugs.

Load balancers

Ideally, a retried request should be routed to a different backend instance than the one that just failed, since the failure might be specific to that instance. Client libraries that support “retry with a different endpoint” behaviour are noticeably more effective than those that simply reconnect to the same address.

Connection pools

Retries consume connections from a pool just like original requests do. If a connection pool is small and a burst of retries occurs, the retries themselves can exhaust the pool, causing new requests — not even the retried ones — to fail with “no available connections.” Sizing connection pools with retry traffic in mind is essential.

Caching as a retry safety valve

When all retries are exhausted, serving a slightly stale cached response is often far better for the user than a hard error. This fallback pattern, sometimes called “serve stale on error,” pairs naturally with a retry policy: try fresh a few times, then fall back gracefully.

In a microservice architecture, a single user-facing request often fans out into many internal calls. Retry policy design here requires thinking about the whole call graph, not just one hop.

Where should retries live?

The general guidance is: retry as close to the failure as reasonably possible, and be conservative about retrying at higher layers that already sit above a lower layer that retries. A common, safer pattern is to retry only at the edge (API gateway, for a limited set of safe / idempotent operations) and at the final hop right before the resource (e.g., the service that talks directly to a database or third-party API) — while intermediate layers simply propagate failures upward without adding their own retry multiplier.

Retries and distributed tracing

Modern tracing systems (like OpenTelemetry) can visualise every attempt within a trace, showing exactly where and why retries happened across a multi-service call. This turns what used to be a mystery — “why did this request take 4 seconds?” — into a clear, inspectable timeline.

Example: Spring Retry annotation in a microservice client

@Service
public class InventoryClient {

    @Retryable(
        value = { TimeoutException.class, HttpServerErrorException.class },
        maxAttempts = 4,
        backoff = @Backoff(delay = 200, multiplier = 2, random = true)
    )
    public InventoryResponse checkStock(String sku) {
        return restTemplate.getForObject(
            "http://inventory-service/stock/" + sku,
            InventoryResponse.class
        );
    }

    @Recover
    public InventoryResponse fallback(Exception e, String sku) {
        // Called only after all retry attempts are exhausted
        return InventoryResponse.unknown(sku);
    }
}

This declarative style, common in Spring-based Java microservices, separates the retry configuration from the business logic entirely — the checkStock method does not need to know retries are happening at all.

A handful of retry-adjacent patterns keep showing up because they answer recurring problems well — and a handful of anti-patterns keep causing incidents because they ignore those same problems.

Established patterns

Anti-patterns to avoid

A short, portable checklist for choosing retry settings on purpose, plus a worked example that shows how to derive them from real latency data instead of copying defaults.

Best practices checklist

• Set a per-attempt timeout and an overall deadline for the whole retry sequence, so a slow dependency can never stall a caller indefinitely.
• Classify errors explicitly — maintain an allowlist of retryable conditions rather than retrying “everything except a few exceptions.”
• Use exponential backoff with jitter as the default; only use fixed delay for extremely low-stakes, low-volume calls.
• Cap the maximum backoff delay so waits do not grow unboundedly on later attempts.
• Pair retries with circuit breakers for calls to external or historically unreliable dependencies.
• Make retries visible: log them, count them, and alert on abnormal retry rates.
• Test retry behaviour deliberately — simulate timeouts and 5xx errors in staging to confirm the policy behaves as designed.

A worked example: tuning a retry policy from scratch

To make all of this concrete, let us walk through how a team might actually derive retry settings for a real endpoint, rather than picking numbers arbitrarily.

1. Measure baseline latency. Suppose the endpoint’s p50 latency is 40ms and its p99 is 250ms under normal conditions. A per-attempt timeout much below the p99 (say, 150ms) would start timing out healthy-but-slow requests, generating unnecessary retries.
2. Set the per-attempt timeout above the healthy p99, with some margin — e.g., 400ms — so only genuinely stuck or failing calls trigger a retry.
3. Decide an acceptable worst-case total latency. If the calling code can tolerate at most 2 seconds of added delay on failure, that constrains how many attempts and how much backoff you can afford.
4. Choose an initial backoff and multiplier that fit inside that budget: 200ms initial, ×2 multiplier, gives roughly 200ms, 400ms, 800ms for three retries — a total of about 1.4 seconds of waiting on top of the attempt timeouts, comfortably under the 2-second budget.
5. Add jitter (e.g., full jitter on the computed delay) to avoid synchronised retries across clients.
6. Set max attempts to 4 (1 original + 3 retries), matching the backoff schedule above.
7. Validate under load by deliberately injecting failures in a staging environment and watching the retry amplification factor and downstream load — adjusting the numbers if the amplification is higher than expected.

This kind of principled, measured tuning — grounded in actual latency data and an explicit latency budget — produces far more reliable outcomes than copying a default configuration from a library’s documentation and hoping it fits.

Common mistakes

A recurring theme across the industry: the companies with the most reliable systems did not just add retries — they paired retries with idempotency guarantees, circuit breakers, and careful observability.

A recurring theme across all of these: the companies with the most reliable systems did not just add retries — they paired retries with idempotency guarantees, circuit breakers, and careful observability. Retries alone are a partial solution; retries as part of a broader resilience strategy are what actually keeps large-scale systems dependable.

Short, opinionated answers to the retry-policy questions that come up most often in interviews, design reviews, and post-incident reviews.

Should I always retry failed requests?

No. Only retry failures that are genuinely transient and operations that are safe to repeat (idempotent, or protected by an idempotency key). Permanent errors like invalid input or authorisation failures should never be retried automatically.

How many times should I retry?

There is no universal number, but 3–5 attempts is a common starting point for most internal service calls, tuned down for latency-sensitive user-facing paths and tuned up (with a circuit breaker as a safety net) for background or batch processing.

What is the difference between a retry policy and a circuit breaker?

A retry policy decides how to handle one call’s failure by trying again. A circuit breaker watches many calls over time and decides whether to stop attempting calls to a dependency altogether. They are complementary, not competing, patterns — most production systems use both together.

Is exponential backoff always better than fixed delay?

For anything beyond a trivial, low-volume use case, yes. Exponential backoff scales the wait time to match how long a dependency likely needs to recover, and combined with jitter it avoids the synchronised retry storms that fixed-delay retries are prone to.

Can retries make things worse?

Yes — this is one of the most important lessons in this topic. Poorly tuned retries (no backoff, no jitter, no maximum, no circuit breaker) can turn a small transient blip into a full cascading outage by adding load to an already-struggling system.

Do I need idempotency keys if my operation is already idempotent?

If the operation is truly idempotent by nature (like a GET or a PUT that fully overwrites a resource with the same data), you generally do not. Idempotency keys matter most for operations like “create” or “charge” that would otherwise produce a new side effect on every call.

Should retries be configured per-service or globally for the whole system?

Both, at different levels. A sensible default policy (a base delay, a multiplier, a max attempt count) makes sense as a global fallback, but individual services should be free to override it based on their own latency characteristics and criticality. A payment service and a low-priority analytics-ingestion service have very different tolerances for added retry latency and load.

What is a retry budget, and how is it different from max attempts?

Max attempts limits one request’s retries in isolation. A retry budget limits retries across the whole system: for example, “retries may never exceed 10% of total request volume in any rolling one-minute window.” Once that budget is used up, new retry attempts are rejected outright, even if an individual request has not hit its own max-attempts limit yet. This protects the system as a whole from a scenario where many individual requests are each within their own limits, but collectively they are still overwhelming a struggling dependency.

How do retries interact with distributed transactions or sagas?

In workflows built from multiple steps across services (often coordinated with the saga pattern), each individual step can have its own retry policy, but the workflow as a whole also needs compensating actions for the case where retries are exhausted partway through — for example, refunding a payment if a later step in an order-fulfilment saga ultimately fails for good. Retries handle transient failure within a single step; sagas handle the broader question of what to undo if a step fails permanently after retries are exhausted.

A retry policy is a deceptively simple idea — “try again if it fails” — that hides real design depth once you consider idempotency, backoff, jitter, error classification, retry budgets, and how retries interact with everything else in a distributed system.

Done well, it turns a system that would otherwise be fragile against everyday network noise into one that quietly absorbs transient failures and keeps working. Done poorly, it can become the very mechanism that turns a minor blip into a cascading outage.