Week 5: How systems fail

Goal: Anticipate failure instead of discovering it.

Live: Incident replay. Facilitators reveal logs, metrics, and alerts step by step. At each stage learners say what they would check next and why.
Async: Run a pre-mortem on your week 4 design: “It’s six months later and this failed. Why?”
Deliverable: Failure-mode map with likelihood, impact, and mitigation for each mode.

Cases for this week

Incident replay · Week 5

Retry storm after a cache outage

The cache recovered within a minute, but the API is still failing 20 minutes later. What is keeping it down, and what would you do first to bring it back?

Intermediate
No AI
Example case

Topics: caching · retries · load

Incident replay · Week 5

Latency creep hours after a routine deploy

Checkout latency is climbing hours after a deploy, while CPU and database load look normal. What is your leading hypothesis, and what do you do in the next 15 minutes?

Advanced
No AI
Example case

Topics: connection-pools · resource-leaks · observability