Functional
- Checkout requests must complete with < 200ms p99 latency
- Order writes must reach a durable relational store
- The fix must keep the existing components' roles intact (this is a live system)
- Idempotent checkout — retried requests must not create duplicate orders
Non-functional
- Restore steady-state error rate to ~0% at 4,000 rps
- Survive the next flash sale (2x traffic) without dropping requests
- Survive a region-A database outage with < 1% error rate
- Total infrastructure cost under $2,000/month
Failure scenarios
- ⚡ Right now (the incident)
- ⚡ Next flash sale (2x traffic)
- ⚡ Region A database outage
Approach guide
- 1.Phase 1 — Triage: What Is Saturated?5m
- 2.Phase 2 — Root Cause: The Capacity Math10m
- 3.Phase 3 — Fix the Write Path15m
- 4.Phase 4 — Availability: Region-A DB Outage15m
- 5.Phase 5 — Incident Retro15m
Full guide on the brief page.
Key numbers
- Total request load (steady state)4,000 rps
- Write load (70% checkout = writes)2,800 rps writes
- Read load (order status, product lookups)1,200 rps reads
- Starting system total checkout-api capacity4,000 rps
- Starting orders-db write capacity2,000 rps