When retries help and when they cause overload
Why the client got no response
Your service may call another service, a dependency: for example, fetching catalog data for a page. Sometimes it responds with errors or too slowly. The client sets a timeout and stops waiting when it expires.
A timeout does not prove that the dependency did not do the work; the response may still be in transit. Here it responds in 200 ms. Waiting only 100 ms prevents success even when the dependency is healthy.
What a retry changes
A retry is another attempt at the same operation after failure. It helps with temporary causes: the next call may succeed. But each retry occupies the dependency again. Under overload, extra calls can make things worse.
Distinguish user operations from attempts. For example, 300 original operations plus 150 retries produce 450 calls. Load amplification is 450 / 300 = 1.5. Each successful operation is counted only once.
Why backoff and jitter are needed
Backoff is the pause before retrying. Exponential backoff grows, for example 500 ms, then 1,000, then 2,000. This gives the dependency time to recover. Retry counts are bounded so operations do not hold resources for too long.
If all clients fail together and wait the same time, they return together. Jitter spreads retries over time. The stand uses three fixed delay groups so identical settings always produce identical results.
How to evaluate retry policy
First compare timeout and response time. Then add retries and compare successful operations, latency and load amplification. More successes at the cost of huge call counts are not necessarily a good solution.
Operations here only read data. Retrying writes needs a separate solution: repeating a payment must not charge twice. “Repeating causes no additional effect” is idempotency. This lab does not model it.
Task
300 operations/s for 40 seconds, then 20 seconds to finish. Dependency response takes 200 ms; every fifth call fails during the first 20 seconds. Choose timeout and retry policy.
Deterministic request cohorts, 100 ms steps, 60s horizon. Original flow lasts 40s. The dependency serves up to 450 attempts/s; step capacity is shared proportionally across cohorts. Transient error share is 20% for the first 20s, then 0%. Each successful operation is counted exactly once.
- Compare the initial timeout with dependency response time.
- Add a retry policy from the palette and adjust attempts, backoff and jitter.
- Balance successful operations, latency and extra load.