Setup
A checkout service runs on 12 instances. Each instance has a database connection pool with a maximum of 30 connections and a 5-second timeout to acquire one.
A routine deploy went out at 10:00. Its main change added promo-code validation to checkout. The deploy passed canary checks and looked healthy for hours.
At 16:30 an alert fires: checkout p99 latency is 4 seconds. Normal is around 300 ms.
Stages
Stage 1: The alert
- CPU on the checkout instances is at 15%.
- Database CPU is at 20%, and the slow query log is empty.
- Error logs show a growing number of
timeout acquiring connection from pool after 5000ms. - No deploys or config changes since 10:00.
Stage 2: Pool metrics
- On the worst instances, all 30 pool connections are checked out.
- On the database side, only 4–6 queries per instance are actually running. Most connections from checkout are idle.
- At 16:40 someone restarted one instance. It was healthy for about 90 minutes, then started degrading again.
- The other instances aren’t degrading at the same rate. Some are almost fine.
Stage 3: Which instances degrade fastest
- The instances degrading fastest are the ones taking traffic from a region running a marketing campaign with promo codes.
- In that region, about 18% of promo codes entered are invalid or expired.
Outcome
The promo validation code took a connection from the pool, and on the “invalid code” path it returned early without releasing it. Each invalid promo code permanently leaked one connection. Instances serving the campaign region leaked fastest and ran out first; every other request then waited 5 seconds for a connection that never came back.
Mitigation that night: turn off promo validation with its feature flag, then do a rolling restart to reclaim the leaked connections. Latency was normal 20 minutes later.
Fix: acquire connections through a scoped helper that always releases them (try/finally, or a with/using block), never by hand. The team also turned on the pool’s leak detection (log a stack trace for any connection held longer than 30 seconds) and added a “time waiting for a connection” metric to the dashboard.
Senior commentary
“Low CPU plus timeouts means the system is waiting, not working. So I stop looking for something slow and start looking for something scarce: connections, threads, file handles, locks.
“Stage 2 is the giveaway. The pool says every connection is in use; the database says most of them are idle. Something took connections and never gave them back. A restart fixing it for 90 minutes confirms it: something that builds up over time and gets cleared by a restart is a leak until proven otherwise.
“‘It was fine for six hours’ doesn’t clear the deploy. Leaks need the right traffic to show up, and the canary didn’t see the campaign region. The uneven degradation across instances tells you the trigger depends on traffic mix, and that points straight at the new code path. For the next 15 minutes I’d buy time with the flag and restarts, then go read the diff for any connection that isn’t released on every path.”
Discussion questions
- What did “the database is mostly idle” rule out, and what did it point to?
- Why didn’t the canary catch this? What would a canary need to catch it?
- Which metric, if it had been on the dashboard, would have made this a 5-minute diagnosis?
- When is a scheduled restart an acceptable mitigation, and when does it hide a problem you need to see?