Setup
500 worker processes each fetch settings from a shared config service every 30 seconds. In normal operation that’s about 17 requests per second.
The config service:
- Can handle about 200 requests per second.
- Has no load shedding. When it’s overloaded, requests queue up and every request gets slower.
Each worker’s client:
- Uses a 2-second request timeout.
- On any failure or timeout, waits 1 second and tries again, with no limit and no jitter, until it succeeds.
The config service is restarted for a deploy and is unavailable for 20 seconds.
Outcome
It stays overloaded until someone intervenes.
During the restart, all 500 workers fall into the retry loop, so the moment the service comes back it receives about 500 requests per second, 2.5× its capacity. Because it doesn’t shed load, requests queue and latency climbs past the 2-second client timeout. Clients give up and retry, but the service keeps working on the requests they’ve abandoned. Almost all of its capacity goes to work nobody is waiting for, so almost nothing succeeds, so nobody leaves the retry loop.
Recovery needs an outside change: restart the workers in batches, add backoff with jitter to the client, or make the service reject excess load quickly so the requests it does accept finish in time.
Senior commentary
“The numbers that matter are the 200 per second capacity and the ‘no load shedding’. Without load shedding, an overloaded server doesn’t do 200 useful requests a second; it does close to zero, because it finishes each one after the client has already left.
“If the service rejected the excess with fast errors instead, it would get through 200 requests a second, those workers would go back to polling every 30 seconds, and it would recover in a few seconds. That’s the version people imagine when they pick ‘it recovers within a minute’. The difference is whether the server spends its capacity on requests someone is still waiting for.”
Discussion questions
- Which detail in the setup decided the outcome for you?
- If you could change just one thing (client timeout, retry delay, jitter, or load shedding), which would you pick and why?
- Compare this with your week 1 answers on similar scenarios. Has your confidence on load questions changed?