Retries make an outage worse
Clients retry failed requests, adding load to a system that is already struggling.
How to approach the problem
Limit retries, add a delay and temporarily stop calls to a failing dependency. Ensure write operations are idempotent separately.
Try it in a lab
Compare successful requests and attempt counts in the retries lab. Then explore how quickly a circuit breaker opens and when it permits a probe.
Labs use educational models. Results do not replace measurements of your system.