loading
loading
You're on call. The flash sale started 10 minutes ago and checkout error rates are climbing past 50%. You inherit the system as it runs — diagnose the bottleneck, fix the architecture under a $2,000/month budget, and be ready for the next sale and a region outage.
Steady traffic
4k rps
Spike
2×
Budget
$2k/mo
Ref. components
6
5 links
How the reference architecture is reasoned out, end to end.
The root cause of the checkout meltdown is a single number: orders-db was provisioned at maxRps=2,000 but the flash sale pushed 2,800 write rps into it — 40% over capacity. This is a write-capacity bottleneck, not a read bottleneck. The cascade is predictable: the database queues writes, the checkout-api connection pool fills up waiting for DB acks, the load balancer queues fill behind the stalled API tier, and the entire stack returns 503s.
The first diagnostic step is to run the sim on the starting system and identify which component shows saturation — orders-db at 140% utilization is the answer. The fix is surgical: upgrade orders-db to handle 6,000+ write rps (headroom for the 2× flash sale spike at 5,600 write rps) and add two additional checkout-api replicas to handle the 8k rps flash sale load (4 replicas × 2,000 = 8,000 rps capacity). A read replica in region-B is also added for availability, but it does not fix the incident — it only helps read availability during an outage.
The idempotency layer (Redis SET NX as the fast path, DB unique constraint as the durability guarantee) prevents duplicate orders when clients retry on 503 errors. This is belt-and-suspenders: Redis NX is sub-millisecond; the DB constraint catches cases where Redis is temporarily unavailable. For the region-A database outage, the write-heavy nature of checkout makes this harder than a read-heavy system.
Reads (GET /orders) can be served from the region-B replica during the outage. Writes (POST /checkout) cannot be completed without a durable write target. The two valid approaches are: (1) queue writes to a durable message queue in region-B for replay when the primary recovers — preserving orders at the cost of delayed confirmation, and (2) return 503 with Retry-After and accept that a 60-second outage means < 1% of daily orders fail — which satisfies the < 1% error rate NFR across a full day.
Both approaches are correct; the key is naming which one you chose and why, and ensuring idempotency on replay if using option 1.
Overall defense
Four checkout-api replicas give 10,000 rps combined capacity, handling both the 4,000 rps steady state (40% utilisation) and the 8,000 rps flash-sale spike (80% utilisation) with no drops. A Redis idempotency cache prevents duplicate order creation under retries without adding DB load. The orders-db primary (10,000 rps cap) sits comfortably at 80% during the spike; the replica in region-b auto-absorbs read traffic during a region-A outage, keeping errors under 1%. Circuit breaking on the api layer prevents a DB degradation event from cascading into a full outage.
The choices that separate a passing design from full credit.
Write capacity, not read capacity, is the bottleneck
Adding a read replica is the most common wrong answer to this incident. Read replicas accept reads — they cannot absorb writes. With a 70% write ratio, 2,800 of the 4,000 rps are inserts into orders-db. The fix must increase write throughput on the PRIMARY. A read replica does not help until after the primary is fixed and you need the region-B outage coverage. Diagnosing the bottleneck type before proposing a fix is the most important skill this challenge tests.
More checkout-api replicas don't help if the DB is the wall
Scaling the API tier when the bottleneck is downstream is the canonical incident misdiagnosis. Adding more checkout-api replicas creates more concurrent DB writers hammering the same saturated database — it makes the problem worse, not better. The sim demonstrates this directly: add 4 more checkout-api replicas without touching orders-db and the error rate does not change. Diagnose the bottleneck (orders-db) before scaling anything.
Idempotency key = Redis SET NX first, DB unique constraint second
The idempotency_key field must be checked BEFORE payment authorization and BEFORE the DB write. The flow is: (1) SET idempotency:{key} {order_id} EX 86400 NX in Redis — if the key exists, return the cached order_id immediately (200, not 201); (2) if Redis SET NX succeeds (key was new), proceed to payment and DB write; (3) the DB UNIQUE constraint on idempotency_key catches the rare case where Redis was unavailable and two requests raced past the NX check. Two independent safety nets: one fast (Redis, sub-ms), one durable (DB).
Flash sale sizing: 2x steady-state, not 1x
The incident recurs if you size to current traffic. The NFR is "survive the next flash sale (2x traffic)." At 8,000 rps with 70% writes = 5,600 write rps. Add 20% headroom = size orders-db for 6,720 write rps. The checkout-api needs 4 replicas × 2,000 rps = 8,000 rps capacity. Every capacity decision must be made against the flash sale peak, not the 4k rps steady state — otherwise you are sizing to the incident that already happened, not the next one.
What most candidates get wrong on this challenge.
Adding a read replica to fix the write bottleneck — read replicas accept reads, not writes. The 2,800 write rps overloading orders-db cannot be rerouted to a replica. This fix addresses the wrong axis entirely and the error rate stays at 52% after adding it.
Scaling checkout-api replicas before fixing orders-db — more API replicas with a saturated downstream just create more concurrent writers hammering the same wall. The DB is the bottleneck; fix it first, then verify the API tier can handle the increased throughput.
Charging payment before checking idempotency_key — if the payment succeeds but the DB write fails (e.g., due to saturation), a client retry creates a second payment charge. The idempotency check must happen first, before any external side effects.
Sizing the fix for 4,000 rps steady state instead of 8,000 rps flash sale — the incident repeats at the next flash sale. Every component must be sized to the 2x spike defined in the NFR, not the current traffic level.
Expand each criterion to see the exact bar.
Full credit requires: orders-db upgraded to at least 3,000 write rps (clearing the 2,800 write rps load), steady-state error rate at 0% in the sim. The upgrade must be argued with the bottleneck math: orders-db at maxRps=2000 receiving 2,800 write rps = 40% over capacity. Partial credit if errors are reduced to < 2% but the root cause math is missing.
Full credit requires: system handles 8,000 rps (2× flash sale) with 0% errors, no component above 90% utilization. orders-db must be sized for at least 5,600 write rps (8k × 70%); checkout-api must have enough replicas for 8k total throughput. The flash sale sizing math must appear in the justification. Partial credit if the spike produces < 2% errors but the sizing is not explicitly argued.
Full credit requires: a read replica in region-B on the canvas, GET /orders served from replica during outage, and a credible story for POST /checkout writes (either queue to region-B for replay with idempotency, or graceful 503 with Retry-After, justified as < 1% of daily orders). Partial credit if replica is present but write availability is unaddressed.
Full credit requires total cost under $2,000/mo with each component size tied to a specific bottleneck number. Reference fix: DB upgrade ($500) + replica ($400) + 2 more checkout-api ($400) + LB ($100) = ~$1,400/mo. Throwing a db.r8g.8xlarge at the problem without justifying why that size is needed loses points for over-provisioning.
Full credit requires all three retro sections argued with numbers: (1) root cause with utilization percentage ("orders-db at 140% of capacity, receiving 2,800 write rps against 2,000 rps limit"), (2) fix with target metrics ("upgraded to 6,000 rps capacity, added 2 checkout-api replicas for 8,000 rps total"), (3) systemic prevention measure with a mechanism ("pre-sale load test at 2× baseline" or "autoscale rule at 80% DB connection pool utilization"). Vague answers like "database was slow" score zero.
The same phase-by-phase guide shown during solving — with answers.
Phase 1 — Triage: What Is Saturated?
Before touching anything, identify the exact component that's causing failures. In the starting system, only one thing can fail at 4k rps — and the sim will show you which component is at 100%+ utilization. Name it before proposing a fix.
Before touching anything, identify the exact component that's causing failures. In the starting system, only one thing can fail at 4k rps — and the sim will show you which component is at 100%+ utilization. Name it before proposing a fix.
Run the sim on the starting system. Which component shows saturation?
show answerhideThe checkout-api has 2 replicas × 2,000 maxRps = 4,000 total rps capacity. That's exactly at 100% utilization — no headroom. The orders-db has maxRps=2,000 but receives 4,000 rps (all checkout traffic, since every checkout writes). Orders-db is at 200% utilization — this is the primary bottleneck.
Why is 50% error rate happening NOW when traffic hasn't changed since launch?
show answerhideFlash sales create a correlated burst: everyone tries to checkout at the same moment. The system was fine at 2k rps (50% utilization), got slammed to 4k rps at sale start, and the orders-db — already the weakest link — immediately saturated. Orders-db backpressure cascades to checkout-api (connection pool full), which cascades to the LB (queue backed up), which starts returning 503s.
Is the problem write capacity, read capacity, or connection limits?
show answerhideWrite capacity. 70% of 4k rps = 2,800 writes/s to the orders-db, but orders-db.maxRps = 2,000. You're asking a database to absorb 40% more writes than it's sized for. The fix is write capacity, not read replicas (read replicas help reads; they don't absorb writes).
Deliverable
One sentence in your overall defense: "The bottleneck is orders-db at [maxRps] with [write_rps] write rps — [%] over capacity."
Phase 2 — Root Cause: The Capacity Math
Quantify exactly what's broken. The incident retro will be graded on whether you can explain the root cause with numbers, not just "the database was overloaded."
Quantify exactly what's broken. The incident retro will be graded on whether you can explain the root cause with numbers, not just "the database was overloaded."
What write rps is the orders-db receiving vs. its capacity?
show answerhide4,000 total rps × 70% write ratio = 2,800 write rps. orders-db maxRps = 2,000. That's 800 rps of unsatisfied write demand, queuing into backpressure, exhausting the checkout-api's DB connection pool. Result: checkout-api blocks, LB queues fill, 503s cascade.
Why didn't the checkout-api scaling (2 replicas) prevent this?
show answerhideAdding more app server replicas doesn't help when the bottleneck is DOWNSTREAM. More checkout-api instances just create more concurrent writers hammering the same saturated DB. You can scale checkout-api to 100 replicas — it won't matter if orders-db can only absorb 2,000 writes/s. This is the most common incident misdiagnosis: "scale the API tier" when the DB is the wall.
What would the flash sale do to a fixed system?
show answerhideFlash sale = 2× traffic = 8,000 rps. Write load = 5,600 write rps. Your fix must handle this, not just 4,000 rps. Size orders-db for 6,000+ write rps with headroom, or add write sharding/queue buffering.
Deliverable
A written root cause in your overall defense: component, utilization percentage, the math showing why it failed, and the cascade chain.
Phase 3 — Fix the Write Path
Fix the orders-db bottleneck without breaking the live system. The constraint: keep existing components — you're modifying, not replacing. Every option has a trade-off you must name.
Fix the orders-db bottleneck without breaking the live system. The constraint: keep existing components — you're modifying, not replacing. Every option has a trade-off you must name.
What are your options to increase DB write throughput?
show answerhideOption A: Vertical scale — upgrade orders-db to a larger instance (higher maxRps). Fastest fix, no code change. Option B: Write queue — put an async queue between checkout-api and orders-db (absorbs bursts but adds latency + complexity). Option C: Write sharding — partition orders-db by user_id or created_at (complex, not needed here). Option D: Connection pooling (PgBouncer) — doesn't increase DB capacity, but reduces connection overhead. For this incident, Option A is the surgical fix.
If you choose vertical scale, what instance size do you need?
show answerhideFlash sale peak = 5,600 write rps. With 2× headroom = 11,200 rps capacity needed. A db.r6g.2xlarge handles ~10,000 rps at $480/mo — within the $2,000 budget. Size for the flash sale, not steady state. Alternatively: db.r6g.xlarge at ~6,000 rps ($300/mo) + 2× checkout-api headroom gets you to the flash sale with ~7% buffer.
Do you need to scale checkout-api too?
show answerhideAt 8,000 rps (flash sale), 2 replicas × 2,000 = 4,000 capacity — only 50% of needed throughput. You need 4 replicas × 2,000 = 8,000 rps capacity. Or increase maxRps per instance to 4,000 and keep 2. The DB fix is required; the API fix is also required for the flash sale scenario.
Deliverable
Updated canvas: upgraded orders-db (higher maxRps) + 2 additional checkout-api replicas (or increased per-instance maxRps). Run sim and verify both scenarios (steady-state and flash sale) show 0% errors.
Phase 4 — Availability: Region-A DB Outage
Design for the region-A outage scenario: orders-db goes down. For a WRITE-HEAVY checkout system, this is harder than the URL shortener's outage (where cache absorbed 95% of reads). Here, writes must land somewhere durable.
Design for the region-A outage scenario: orders-db goes down. For a WRITE-HEAVY checkout system, this is harder than the URL shortener's outage (where cache absorbed 95% of reads). Here, writes must land somewhere durable.
What happens to checkouts when orders-db (region-A) is down?
show answerhideEvery POST /checkout requires a write. If the primary DB is down, you CANNOT complete a checkout without losing order data. Options: (A) Accept errors for the 60s outage — < 1% of daily orders (the NFR allows < 1% errors), (B) queue writes to a secondary DB in region-B, (C) failover to a hot standby in region-B. Option A sounds bad but IS < 1% errors across a day. The NFR says < 1% error rate, not zero.
Can a read replica in region-B absorb the outage?
show answerhideA read replica accepts reads, not writes. For the < 1% error rate NFR to be met via a replica, you'd need to: (A) accept reads from the replica during outage (GET /orders works), (B) reject writes with a queued retry (POST /checkout returns 503 with Retry-After, queues to a durable message queue in region-B for replay when primary recovers). This is the safest write-heavy availability pattern.
What is 'durable' for checkout writes during an outage?
show answerhideIf you queue checkouts to Kafka/SQS in region-B and replay them when the primary comes back, you must handle: (A) the customer was charged but no order appeared yet — need idempotency on replay, (B) inventory may have changed during the 60s — need conflict resolution. The idempotency_key in the DB schema handles (A).
Deliverable
A Postgres read replica in region-B on the canvas. Justification must explain: GET /orders reads via replica (stays up during outage), POST /checkout writes either queue to region-B or fail with graceful 503 (< 1% error rate across the day). Both are valid — name which you chose and why.
Phase 5 — Incident Retro
Write the incident retro in your overall defense. A PRR retro has three mandatory sections: what happened (root cause + impact), what changed (the fix, with numbers), and what prevents a repeat (the systemic change). Vague retros lose points.
Write the incident retro in your overall defense. A PRR retro has three mandatory sections: what happened (root cause + impact), what changed (the fix, with numbers), and what prevents a repeat (the systemic change). Vague retros lose points.
What is the precise root cause, in one sentence with numbers?
show answerhideModel answer: "orders-db was provisioned at 2,000 rps write capacity but received 2,800 rps at the flash sale peak (40% over capacity), causing connection pool exhaustion that cascaded to checkout-api (100% utilization) and LB backpressure, resulting in 52% error rate."
What systemic change prevents a repeat?
show answerhideOptions: load testing before each flash sale (detects the capacity gap before production), autoscale rules on orders-db CPU/connections that trigger a vertical scale or read-offload, a circuit breaker on checkout-api that queues rather than errors when DB is saturated, or a write queue (SQS) that absorbs burst writes and smooths DB load. Name one and argue why it specifically prevents THIS incident.
What monitoring would have caught this BEFORE the sale?
show answerhidePostgres connection pool usage (alert at 80% — that's the signal that a DB is approaching saturation), orders-db write IOPS (alert at 60% of max — gives time to act), checkout-api error rate (alert at 1% — too late for this incident, should have been caught pre-sale). Pre-sale load test simulating 2× base traffic is the real answer.
Deliverable
Overall defense that includes: root cause with utilization number, fix description with target metrics, and one systemic prevention measure. This is what gets graded on the justification-quality axis.
Ready to test your design?
Open the canvas, place your components, and run the failure scenarios to get graded.