The Retry Storm
Four clients, one server and a brief outage. How the clients retry decides whether the server recovers when it comes back, or stays buried.
Four clients, one server and a brief outage. How the clients retry decides whether the server recovers when it comes back, or stays buried.
Experiment 02 · Reliability
Bar: calls arriving each tickFilled: calls servedDashed: room for 9Shaded: server down
The server has room for 9 calls a tick. Overloaded, it wastes effort on calls that time out, and only 4 complete. Each call is tried at most four times.
Four clients, one server with room for 9 calls a tick. Clients retry at once when a call fails.
Tick 1: 4 calls, all within capacity.
Tick 2: 4 calls, all within capacity.
Tick 3: 4 calls, all within capacity.
Tick 4: the server is down. All 4 calls fail.
Tick 5: the server is down. All 8 calls fail, 4 of them already retries.
Tick 6: the server is down. All 12 calls fail, 8 of them already retries.
Tick 7: the server is down. All 16 calls fail, 12 of them already retries.
Tick 8: 16 calls arrive, 12 of them retries, against room for 4. 12 are turned away.
Tick 9: 16 calls arrive, 12 of them retries, against room for 4. 12 are turned away.
Tick 10: 16 calls arrive, 12 of them retries, against room for 4. 12 are turned away.
Tick 11: 16 calls arrive, 12 of them retries, against room for 4. 12 are turned away.
Tick 12: 16 calls arrive, 12 of them retries, against room for 4. 12 are turned away.
Tick 13: 16 calls arrive, 12 of them retries, against room for 4. 12 are turned away.
Tick 14: 16 calls arrive, 12 of them retries, against room for 4. 12 are turned away.
Tick 15: 12 calls arrive, 12 of them retries, against room for 4. 8 are turned away.
Tick 16: 8 calls (8 retries), all within capacity.
Tick 17: 0 calls, all within capacity.
Tick 18: 0 calls, all within capacity.
Tick 19: 0 calls, all within capacity.
Tick 20: 0 calls, all within capacity.
| Policy | Peak calls | Given up | Settled |
|---|---|---|---|
| Retry at once | 16 | 4 | +8 ticks |
| Backoff | 16 | 0 | +8 ticks |
| Backoff + jitter | 15 | 1 | +3 ticks |
Each tower is a client sending one call per tick. The server can take nine calls a tick, more than the four it normally receives. Then it goes down. Every call in the outage fails, and every failed call is retried.
When the server comes back it faces the normal traffic plus every retry. Past its capacity it slows down and wastes effort on calls that will time out, so fewer complete. That is the trap: the retries meant to recover from the outage are what keep the server overloaded.
Time moves in ticks, every client behaves identically and the overload penalty is a fixed number. Real systems add timeouts, circuit breakers, retry budgets and load shedding. Jitter here comes from a seeded sequence, so every run replays exactly.