Incident: The Retry Storm That Took Down Payments

The retries arrived immediately (retry-delay=0ms). At the moment the network recovered, the 9,600 failed requests were all being retried simultaneously — generating 4× the normal load (8,000 rps). This 8,000 rps is above payment-api's 5,000 rps capacity. So the payment-api was now overloaded by the retries, causing new failures, which triggered new retries, keeping the load at 8,000 rps. The network hiccup set a self-sustaining load amplification loop in motion.

retry-count=3, retry-delay=0ms. Each failed request generates 3 retries fired immediately. Total attempts per failed request = 1 (original) + 3 (retries) = 4. All 4 arrive at the same instant. During the outage, 100% of 2,000 rps is failing (because the API is saturated). So the effective load = 2,000 × 4 = 8,000 rps — 60% above the 5,000 rps capacity.

Any mechanism that spreads the retries over time: (1) exponential backoff — each retry fires later than the last, spreading 9,600 simultaneous retries over 700ms; (2) rate limiting — caps retries per user so the total load can't exceed capacity; (3) a circuit breaker on the client — after N consecutive failures, stop sending until the circuit resets. Without any of these, the system has no mechanism to self-regulate.

Exponential backoff: retry 1 waits 100ms, retry 2 waits 200ms, retry 3 waits 400ms. Total spread: ~700ms. Instead of 9,600 retries arriving in 0ms, they arrive spread over 700ms. Peak retried load drops from 8,000 rps (9,600 retries / 0ms window) to roughly 9,600 / 0.7s ≈ 13,700 retries/s — wait, that's still 13k. The key insight: retries from retry 1 (100ms after the hiccup ended) are spread over multiple cycles, and the load decreases each cycle as successful retries drain the queue.

Without jitter, every client that failed at T=0 retries at T=100ms (all simultaneously), then at T=300ms (all simultaneously), then at T=700ms (all simultaneously). You get three synchronized spikes instead of one continuous spike — the thundering herd just has a different shape. With jitter, each client adds a random delay to its backoff (e.g., retry 1 at 100ms ± random(0, 100ms)), spreading retries uniformly over the backoff window. No synchronized spikes. Full jitter formula: sleep = random(0, min(cap, base × 2^attempt)).

3 retries with exponential backoff is reasonable for transient network errors. However, payment failures require user feedback — infinite retry without surfacing the failure to the user is wrong. After 3 retries, return a failure to the user with a message like "payment is taking longer than expected — check your payment history." The idempotency_key means a successful eventual delivery won't double-charge. Retry budget: total retry time should stay under the user's patience threshold (~30s).

Error rate threshold: if more than 50% of DB calls in the last 10s failed (timeout or error), open the circuit. Latency threshold: if p99 DB latency exceeds 2s, open. Once open, all DB calls fail immediately with 503 — no waiting, no connection pool usage. The threshold window (10s) must be short enough to react to an incident quickly but long enough to avoid flapping on transient errors.

After the circuit has been open for 30 seconds, transition to half-open: allow exactly one probe request to pass through to the DB. If the probe succeeds (DB responded in < 500ms), close the circuit and resume normal operation. If the probe fails, re-open for another 30 seconds. Without half-open, you need a human to manually close the circuit. With half-open, the circuit self-heals once the DB recovers — no pager needed if the DB recovers within its normal self-healing window.

In Redis (circuit:payment-db key). Why not in-memory? If each payment-api instance maintains its own in-memory circuit state, they may be in different states — one instance trips the circuit, the others don't know and keep hammering the DB. Shared state in Redis means all payment-api instances see the same circuit state. Use Redis SET EX for the open state (auto-expires after 30s) and SET for closed/half-open.

A retry budget is a cap on the percentage of traffic that is retries, measured at the API gateway level. Example: if more than 20% of requests in the last 10s are retries (identified by idempotency_key seen before), the gateway rejects them with 429. This caps the retry load multiplier at 1.25x regardless of client retry configuration — even if clients retry 10 times, only 25% extra load reaches the service. Implement at the LB or API gateway, not in individual services.

A request queue (SQS/in-memory) absorbs bursts but adds latency. For payments (latency- sensitive), a queue is only appropriate if: (1) the burst is temporary (< 60s) and (2) users are informed that processing is async. For the retry storm scenario, the burst lasted 45 minutes — a queue would have accumulated millions of backlogged payments and taken hours to drain. Better to reject excess with 429 + Retry-After and let clients self-regulate with proper backoff. Queuing is not a substitute for backoff — it delays the problem.

Three signals that fire within the first 60 seconds: (1) payment-api error rate alert (> 1% for > 30s), (2) payment-api request rate alert (> 120% of baseline), (3) payment- db connection pool utilization alert (> 80%). If any of these alert to an on-call engineer within 60 seconds of the hiccup, they can manually open the circuit breaker or add capacity before the feedback loop stabilizes. The 45-minute outage is a monitoring failure as much as an architecture failure.