Evolution: Scale a Photo Feed Through Three Growth Stages

1,800 reads/s x 5ms/query = 9 concurrent queries on average. Postgres handles 500+ concurrent connections -- this is ~2% utilization. Fine. Stage 1 doesn't need Redis for the feed. One feed-api instance at maxRps=2,000 + social-db at maxRps=3,000 handles this comfortably within the $1k budget.

Photos go to object storage (S3/GCS). The feed API returns JSON with post metadata + pre-signed thumbnail URLs. The API never serves binary data. At Stage 1, pre-signed URLs are generated on the fly per request (< 1ms). Photo thumbnails are pre-generated by a background job after upload.

posts: (user_id, created_at DESC) for get-posts-by-user. user_follows: (followee_id) for fan-out queries. Adding the followee_id index at Stage 2 on a 2B row table requires a multi-hour lock. Adding it at Stage 1 on a 10M row table takes seconds. The schema decisions made now are the ones you'll live with at Stage 3.

Feed pages are personalized (unique per user) -- a CDN keyed only on the URL (/feed) would serve user A's feed to user B. You'd need to key on the session cookie, which means effectively 0% CDN hit rate. CDN doesn't help for personalized feeds. CDN DOES help for individual post thumbnails and photos -- those are shared content.

The feed INDEX per user -- a list of post IDs in order, not the full post data. Post metadata (image URL, caption, like count) is shared across all viewers who see that post, so it CAN be cached in shared storage. Two-layer strategy: feed:{user_id} ZSET (personalized index) + post:{id} HASH (shared metadata). Feed read = 1 ZREVRANGE + N HGETALL, all in Redis, zero Postgres on the hot path.

On POST /posts: query user_follows WHERE followee_id = $poster to get all followers, then ZADD post_id to each follower's feed:{follower_id} ZSET. This is fan-out on write. At Stage 2 with avg 200 followers: 200 ZADD operations per post. At 200 writes/rps x 200 ZADD = 40,000 Redis writes/rps from fan-out -- still fine for Redis. The celebrity case (5M followers) is where this breaks at Stage 3.

Fan-out on write: enumerate 5M followers then ZADD to each feed. At 100k ZADD/s per Redis instance, this takes 50 seconds per celebrity post. During those 50 seconds, the feed staleness NFR (5s) is violated for all 5M followers. If 10 celebrities each post simultaneously, the fan-out queue backs up by 50M operations and never catches up.

Split by follower count at a threshold (e.g., 100k followers). NORMAL users (below threshold): fan-out on WRITE as before. CELEBRITY users (above threshold): pull on READ. When the feed builder constructs a feed for a user who follows celebrities, it fetches each celebrity's recent posts from their own ZSET and merges them with the pre-computed feed at read time. This adds 1-2 Redis ZREVRANGE calls per celebrity followed -- usually bounded to < 5 celebrities per user in practice.

The threshold = fan-out worker throughput (ZADD/s) x staleness budget (seconds). If you can do 100k ZADD/s across all workers and the NFR is 5s: max fan-out = 100k x 5 = 500k followers. Set threshold at 500k. Lowering the threshold protects Redis from fan-out storms but increases the number of celebrities needing pull-on-read merging. It's a tuning knob -- size it to your worker capacity, not an arbitrary "famous person" number.

Yes -- fall back to the Stage 1 Postgres query. It's slow (5-10ms) and can't handle 36,000 rps, but during a Redis outage you rate-limit feed requests to what Postgres can absorb (~3,000 rps) and shed the rest with 503 + Retry-After. Users see a "feed updating" indicator (the stale flag) rather than a hard error. A partial feed is better than a 503 -- the API contract already includes the stale field.

Fan-out workers must be able to pause and replay. Use a durable message queue (SQS or Kafka) between POST /posts and the fan-out workers. During Redis outage: workers retry ZADD with exponential backoff, failed operations stay in the queue. When Redis recovers, the queued events replay and feed caches are rebuilt. Without a durable queue, posts written during the outage are silently missing from followers' feeds even after Redis recovers.

social-db primary goes down: writes (POST /posts, follow/unfollow) fail with 503. Feed reads continue -- they hit Redis first, and the Redis cluster is cross-region. Reads that miss Redis fall back to the read replica in region-B. The < 5% error rate NFR covers the outage period: reads stay at ~0% errors, writes fail for the duration. This is correct prioritization -- in a social feed, reads are more valuable than writes during an incident.

Adding Redis at Stage 1 costs $300-600/mo for almost no benefit -- Postgres serves 1,800 rps at 2% utilization. The Stage 2 Redis purchase deferred saves $600/mo for 2 months = $1,200 saved before it's needed. Building the fan-out worker at Stage 1 is a full engineering quarter on infrastructure that isn't needed for 6 months.

The schema and indexes. Adding the (followee_id) index to a 2B-row table at Stage 2 requires a full table scan -- hours of locking in production. Adding it at Stage 1 on 10M rows takes seconds. The pagination cursor format (opaque cursor vs. page number) is a client contract -- changing it at Stage 2 breaks all existing mobile clients. The media upload flow (pre-signed URL to object storage) -- migrating binary data later means touching every stored photo.

CDN hit rate for photo thumbnails. At 36k reads/rps with 80KB average thumbnail: 36,000 x 80KB x $0.085/GB CDN egress = ~$9,000/mo IF all requests are cache misses. At 99% CDN hit rate: $90/mo egress. Every 1% improvement in CDN hit rate saves ~$90/mo. Use aggressive Cache-Control headers (max-age=31536000 for immutable photos) and set a thumbnail CDN cache hit rate alert as a cost KPI.