Temporarily isolating an unavailable service
Why someone else’s failure affects your service
Suppose your service calls an external service. While it works, calls are useful. When unavailable, new calls keep failing. In real systems, waiting for these responses can occupy connections and handlers needed for other work.
A circuit breaker protects the path to a dependency. It detects errors and temporarily blocks calls. This limits useless calls, but neither fixes the dependency nor makes rejected requests successful.
The breaker's three states
In the closed state, requests pass normally. When errors reach the threshold, the breaker switches to open. New requests fail fast without reaching the dependency. The names come from electrical circuits: a closed circuit conducts current, an open one does not.
After a pause, half-open mode starts and admits a limited number of calls. A successful probe restores normal traffic; a failed probe restarts the pause. Here the threshold counts consecutive failed seconds and half-open admits up to ten calls per second.
Choosing when to trip and recover
Imagine a dependency is unavailable for ten seconds. Without protection, thousands of useless calls could reach it during that time. Fast disconnection reduces them, and periodic probes detect recovery.
Too high a threshold keeps traffic flowing for too long. Too long a pause delays return after recovery. Too short a pause repeatedly probes a service that is still down. Compare these effects separately.
What counts as success in the lesson
The dependency is unavailable from second 10 until the start of second 25. Watch how many calls reach it during the outage and when full traffic returns. The analysis lets you follow state changes second by second.
The overall failure rate alone does not explain protection quality: success is impossible during the outage here. A separate metric therefore shows successful requests outside the outage. Good protection limits useless work and restores useful work promptly.
Task
The dependency is unavailable from second 10 to 25. Stop calls during failure and restore traffic after a successful probe. Fast failure protects the system but is not a successful request.
60 one-second steps, steady flow. The dependency serves 500 req/s outside outage 10 ≤ t < 25; capacity is zero during it. Closed counts consecutive failed seconds. Open rejects calls before they reach the dependency. After cooldown, half-open admits up to 10 probes/s; the rest fail fast.
- Observe dependency calls during the outage without protection.
- Add a circuit breaker from the panel and explore trip threshold and cooldown.
- Limit calls during failure while restoring traffic promptly.