loading
loading
A seed-funded photo feed app just raised its Series A. Your job is to design a system that works at 2,000 rps today, survives a 5x growth to 10,000 rps after the press cycle, and is architecturally ready for the 40,000 rps a Series B brings -- all within per-stage infrastructure budgets. 90% of traffic is feed reads. The personalization requirement changes everything about what you can cache.
Steady traffic
2k rps
Spike
3×
Budget
$8k/mo
Ref. components
7
6 links
How the reference architecture is reasoned out, end to end.
The three-stage evolution of a photo feed system teaches a fundamental truth: the same problem — serving a user's feed — requires a completely different architecture at each order of magnitude. At Stage 1 (2k rps), the naive Postgres query (JOIN posts with user_follows, order by created_at, limit 20) runs at roughly 2% of DB capacity. No Redis needed.
The $1,000/month budget is easily satisfied with a single feed-api instance and a single social-db. The critical work at Stage 1 is not performance — it's schema hygiene: add the (followee_id) index to user_follows now, on a 10M-row table, because adding it at Stage 2 on a 2-billion-row table requires hours of locking. At Stage 2 (10k rps), the same Postgres query saturates.
9,000 read rps × 5ms per query = 45 concurrent queries, well within Postgres limits, but the query itself is a multi-table JOIN with a full sort — and at 10k rps you have ~3x more queries than Stage 1. The fix is a Redis ZSET feed index per user: feed:{user_id} holds the 100 most recent post IDs in score order. A feed read becomes one ZREVRANGE (1ms) plus N HGETALL calls for post metadata (also in Redis).
Zero Postgres on the hot path. Fan-out on write: when a user posts, the fan-out worker enumerates their followers (from user_follows) and ZADDs the post_id to each follower's ZSET. At an average 200 followers, this is 200 ZADD operations per post — trivial for Redis.
At Stage 3 (40k rps), the fan-out on write approach breaks for celebrity accounts. A user with 5 million followers posting triggers 5 million ZADD operations — taking 50 seconds at 100k ZADD/s worker throughput, violating the 5-second staleness NFR for all 5 million followers. The solution is hybrid fan-out: users below the follower threshold (computed as worker ZADD/s × staleness_budget_seconds = 100k × 5 = 500k followers) continue with fan-out on write; celebrity users above the threshold use pull-on-read.
At feed construction time, for each celebrity a user follows, the feed builder fetches that celebrity's recent posts from their dedicated ZSET and merges them at read time. CDN handles photo thumbnails (immutable, Cache- Control max-age=31536000) — a 99% CDN hit rate on thumbnail delivery is the largest cost lever at Stage 3, reducing CDN egress cost from ~$9,000/mo to ~$90/mo. Graceful degradation is the availability story: if Redis cluster fails, the feed falls back to the Stage 1 Postgres query, rate-limited to what Postgres can absorb (~3,000 rps).
The API response includes a stale flag so clients show a "feed updating" indicator rather than a hard error. Fan-out workers use a durable message queue (SQS/Kafka) between POST /posts and the workers, so posts written during a Redis outage are queued and replayed when Redis recovers — no silent data loss. When social-db primary fails, reads continue from the region-B replica (via Redis cache, then replica on miss); writes fail with graceful 503 for the duration.
Overall defense
The design scales across four growth stages by stacking cache layers. Stage 1 (2,000 rps): three feed-api replicas (9,000 rps cap) handle the load trivially. Stage 2 (10,000 rps): api replicas still have 90% headroom. Stage 3 (40,000 rps): CDN at 85% hit ratio reduces api-bound traffic to 6,000 rps; the Redis feed cache at 90% hit ratio further reduces social-db load to ~600 rps. Stage 4 (region-A outage): the social-db replica in region-b absorbs read traffic automatically via same-kind failover — no manual intervention required. Each layer is independently scalable and the design has no cross-region write dependency for reads.
The choices that separate a passing design from full credit.
Feed cache is a ZSET of post IDs, not cached HTML
Caching full feed JSON (with post content, like counts, author metadata) means every like event, every post update, every profile change requires invalidating millions of per-user caches. The invalidation cost is unbounded. The correct unit to cache is the feed INDEX: a ZSET of post IDs in timestamp order per user. Post metadata (image URL, caption, like_count) is cached separately as shared HASH entries (post:{post_id}) accessible by all viewers of that post. Feed read = 1 ZREVRANGE + N HGETALL, all Redis, all fast. Invalidation is surgical: a like event only updates one post:{id} HASH, not millions of feed caches.
CDN does not help personalized feeds
A CDN caches responses keyed on the URL. /feed is the same URL for every user — so a CDN keyed on path alone would serve user A's feed to user B. You could key on the session cookie, but then every user is a cache miss and CDN hit rate is ~0%. CDN is the right answer for shared content: photo thumbnails, post images, static assets. The Redis ZSET IS the personalized cache layer — it is the equivalent of a CDN for private content. The two mechanisms serve different content types and must not be conflated.
The celebrity threshold is computed, not intuited
The fan-out threshold is not "famous people" — it is a function of worker throughput and the staleness NFR. If fan-out workers can do 100k ZADD/s across all workers and the staleness budget is 5 seconds, the maximum fan-out per post is 100k × 5 = 500k followers. Set the threshold at 500k: anyone with more than 500k followers gets pull-on-read. This threshold is a tuning knob, not a fixed number — double the workers and you can double the threshold. The engineering question is how to compute it, not what the "right" number is intuitively.
Redis outage degrades to Postgres, not 503
The stale flag in the GET /feed response is the graceful degradation signal. When Redis is unavailable, the feed-api falls back to the Stage 1 Postgres query — rate-limited to what Postgres can absorb (~3,000 rps from its capacity). Requests above that limit receive 503 with Retry-After. Users with cached feeds from their device/browser see no disruption. The stale: true flag in the response tells the client to show a "feed may be updating" indicator rather than an error. This is a deliberate product decision encoded in the API contract — it must be designed before the incident, not during it.
What most candidates get wrong on this challenge.
Adding CDN for the /feed endpoint — personalized feeds have a ~0% CDN hit rate when keyed on URL alone. CDN is the correct answer for photo thumbnails (shared, immutable), not for per-user feed pages. Conflating CDN caching with personalized feed caching is the most common Stage 2 design error.
Caching full rendered feed JSON instead of the post ID index — any update to a post (new like, caption edit, deletion) requires invalidating every user's feed cache that includes that post. At 10M DAU × average 5 mutual-following relationships per post, a single like can trigger millions of cache invalidations. Cache the index (post IDs), not the content.
Applying fan-out on write to all users regardless of follower count — a celebrity with 5M followers generates 5M ZADD operations per post. At 100k ZADD/s, that takes 50 seconds, violating the 5s staleness NFR. The hybrid fan-out strategy (fan-out for normal users, pull-on-read for celebrities) must be designed before it is needed — not discovered in production when a celebrity joins the platform.
Adding Redis at Stage 1 — Postgres serves Stage 1's 1,800 read rps at roughly 2% utilization. Adding Redis at Stage 1 costs $300-600/mo for zero measurable benefit and introduces operational complexity. The correct decision is to defer Redis until Stage 2 when Postgres actually saturates. Over-engineering for Stage 3 on Day 1 is a cost and complexity violation.
Expand each criterion to see the exact bar.
Full credit requires: Postgres as the sole feed store (no Redis), 0% errors at 2k rps, total cost within $1k/mo, AND no unnecessary components added. The schema must include the (followee_id) index on user_follows — this is the Stage 1 decision that determines whether Stage 2 fan-out is possible without a multi-hour migration. Partial credit if errors are 0% but Redis is added (cost violation and premature complexity).
Full credit requires: Redis ZSET feed cache (feed:{user_id}) present, fan-out on write implemented (ZADD to follower feeds on each post), post metadata in Redis HASH, 0% errors at 10k rps, within $3k/mo. Justification must explain why CDN doesn't help here (personalization) and what the invalidation strategy is (ZADD on new post, not DEL on update). Partial credit if cache is present but Postgres is still the primary read path.
Full credit requires: hybrid fan-out described with the threshold computation (worker ZADD/s × staleness_budget_seconds), pull-on-read for celebrity accounts, CDN for photo thumbnails with Cache-Control strategy, 0% errors at 40k rps, within $8k/mo. The celebrity threshold must be computed from numbers, not asserted intuitively. Partial credit if fan-out is handled but celebrity case is unaddressed (noted as future work doesn't count).
Full credit requires: Redis outage degradation to Postgres described (rate-limited, stale flag), durable message queue between POST /posts and fan-out workers (so fan-out events survive Redis outage), social-db primary outage handled via region-B replica + Redis for reads. Error rate under 5% during region-A outage. Partial credit if replica is present but Redis outage degradation path is not described.
Full credit requires a per-stage narrative with at least one utilization number per transition: "Postgres at Stage 1: 2% utilization (1,800 rps × 5ms = 9 concurrent queries). At Stage 2: 9,000 rps × 5ms = 45 concurrent queries at 300% utilization — Redis needed. Celebrity threshold: 100k ZADD/s × 5s = 500k follower limit." CDN hit rate economics for Stage 3 thumbnails should appear. Partial credit if design choices are correct but justifications lack specific numbers.
The same phase-by-phase guide shown during solving — with answers.
Phase 1 -- Stage 1 Baseline: Make 2k rps Work
Design the simplest possible system that serves 2k rps within a $1,000/month budget. At this scale, Postgres can serve all reads. The goal is a clean foundation -- not over-engineering for Stage 3 on Day 1.
Design the simplest possible system that serves 2k rps within a $1,000/month budget. At this scale, Postgres can serve all reads. The goal is a clean foundation -- not over-engineering for Stage 3 on Day 1.
At 1,800 read rps and a 5ms Postgres query: what is the DB utilization?
show answerhide1,800 reads/s x 5ms/query = 9 concurrent queries on average. Postgres handles 500+ concurrent connections -- this is ~2% utilization. Fine. Stage 1 doesn't need Redis for the feed. One feed-api instance at maxRps=2,000 + social-db at maxRps=3,000 handles this comfortably within the $1k budget.
Where do the photos live, and what does a feed response actually contain?
show answerhidePhotos go to object storage (S3/GCS). The feed API returns JSON with post metadata + pre-signed thumbnail URLs. The API never serves binary data. At Stage 1, pre-signed URLs are generated on the fly per request (< 1ms). Photo thumbnails are pre-generated by a background job after upload.
What indexes must exist on posts and user_follows from Day 1?
show answerhideposts: (user_id, created_at DESC) for get-posts-by-user. user_follows: (followee_id) for fan-out queries. Adding the followee_id index at Stage 2 on a 2B row table requires a multi-hour lock. Adding it at Stage 1 on a 10M row table takes seconds. The schema decisions made now are the ones you'll live with at Stage 3.
Deliverable
Canvas: feed-api (1 replica) + social-db + photo-storage. Verify 0% errors at 2k rps and total spend well under $1k/mo.
Phase 2 -- Stage 2 Cache Strategy: Handle 10k rps
At 9,000 feed reads/rps, social-db is saturated. You need a cache layer -- but what do you cache? The answer isn't obvious because feeds are personalized. Figure out what's cacheable and what isn't before adding Redis.
At 9,000 feed reads/rps, social-db is saturated. You need a cache layer -- but what do you cache? The answer isn't obvious because feeds are personalized. Figure out what's cacheable and what isn't before adding Redis.
Can you cache the feed response in a CDN or reverse proxy?
show answerhideFeed pages are personalized (unique per user) -- a CDN keyed only on the URL (/feed) would serve user A's feed to user B. You'd need to key on the session cookie, which means effectively 0% CDN hit rate. CDN doesn't help for personalized feeds. CDN DOES help for individual post thumbnails and photos -- those are shared content.
What's the unit you CAN cache that makes sense for ALL users simultaneously?
show answerhideThe feed INDEX per user -- a list of post IDs in order, not the full post data. Post metadata (image URL, caption, like count) is shared across all viewers who see that post, so it CAN be cached in shared storage. Two-layer strategy: feed:{user_id} ZSET (personalized index) + post:{id} HASH (shared metadata). Feed read = 1 ZREVRANGE + N HGETALL, all in Redis, zero Postgres on the hot path.
What invalidates the feed cache when a new post is created?
show answerhideOn POST /posts: query user_follows WHERE followee_id = $poster to get all followers, then ZADD post_id to each follower's feed:{follower_id} ZSET. This is fan-out on write. At Stage 2 with avg 200 followers: 200 ZADD operations per post. At 200 writes/rps x 200 ZADD = 40,000 Redis writes/rps from fan-out -- still fine for Redis. The celebrity case (5M followers) is where this breaks at Stage 3.
Deliverable
Canvas: feed-api + Redis cluster + social-db (now mainly for writes and cold-start rebuilds). Verify 0% errors at 10k rps, within $3k/mo budget.
Phase 3 -- Stage 3 Fan-Out: Solving the Celebrity Problem
At Stage 3 (40k rps, 4k post writes/rps), naive fan-out on write breaks for high-follower accounts. Design the hybrid fan-out strategy that handles both regular users (under 100k followers) and celebrities (millions of followers) without grinding fan-out workers to a halt.
At Stage 3 (40k rps, 4k post writes/rps), naive fan-out on write breaks for high-follower accounts. Design the hybrid fan-out strategy that handles both regular users (under 100k followers) and celebrities (millions of followers) without grinding fan-out workers to a halt.
What happens to your fan-out workers when a celebrity (5M followers) posts?
show answerhideFan-out on write: enumerate 5M followers then ZADD to each feed. At 100k ZADD/s per Redis instance, this takes 50 seconds per celebrity post. During those 50 seconds, the feed staleness NFR (5s) is violated for all 5M followers. If 10 celebrities each post simultaneously, the fan-out queue backs up by 50M operations and never catches up.
What is the hybrid fan-out strategy?
show answerhideSplit by follower count at a threshold (e.g., 100k followers). NORMAL users (below threshold): fan-out on WRITE as before. CELEBRITY users (above threshold): pull on READ. When the feed builder constructs a feed for a user who follows celebrities, it fetches each celebrity's recent posts from their own ZSET and merges them with the pre-computed feed at read time. This adds 1-2 Redis ZREVRANGE calls per celebrity followed -- usually bounded to < 5 celebrities per user in practice.
How do you determine the safe celebrity threshold?
show answerhideThe threshold = fan-out worker throughput (ZADD/s) x staleness budget (seconds). If you can do 100k ZADD/s across all workers and the NFR is 5s: max fan-out = 100k x 5 = 500k followers. Set threshold at 500k. Lowering the threshold protects Redis from fan-out storms but increases the number of celebrities needing pull-on-read merging. It's a tuning knob -- size it to your worker capacity, not an arbitrary "famous person" number.
Deliverable
Canvas: async fan-out worker pool + celebrity bypass path in feed builder. Verify 0% errors at 40k rps and describe how a celebrity post is served to their followers within the 5s staleness budget.
Phase 4 -- Graceful Degradation and DB Availability
At Stage 3 the entire feed depends on Redis. A Redis cluster failure means ALL feeds fail simultaneously. Design the degradation modes and the region-A outage plan BEFORE the outage happens.
At Stage 3 the entire feed depends on Redis. A Redis cluster failure means ALL feeds fail simultaneously. Design the degradation modes and the region-A outage plan BEFORE the outage happens.
If Redis cluster is unavailable, can you still serve a feed?
show answerhideYes -- fall back to the Stage 1 Postgres query. It's slow (5-10ms) and can't handle 36,000 rps, but during a Redis outage you rate-limit feed requests to what Postgres can absorb (~3,000 rps) and shed the rest with 503 + Retry-After. Users see a "feed updating" indicator (the stale flag) rather than a hard error. A partial feed is better than a 503 -- the API contract already includes the stale field.
What happens to the fan-out queue during a Redis outage?
show answerhideFan-out workers must be able to pause and replay. Use a durable message queue (SQS or Kafka) between POST /posts and the fan-out workers. During Redis outage: workers retry ZADD with exponential backoff, failed operations stay in the queue. When Redis recovers, the queued events replay and feed caches are rebuilt. Without a durable queue, posts written during the outage are silently missing from followers' feeds even after Redis recovers.
What is the plan when social-db primary fails?
show answerhidesocial-db primary goes down: writes (POST /posts, follow/unfollow) fail with 503. Feed reads continue -- they hit Redis first, and the Redis cluster is cross-region. Reads that miss Redis fall back to the read replica in region-B. The < 5% error rate NFR covers the outage period: reads stay at ~0% errors, writes fail for the duration. This is correct prioritization -- in a social feed, reads are more valuable than writes during an incident.
Deliverable
State your degradation modes in the overall defense: Redis failure path, social-db primary failure path, and the durable queue that prevents silent data loss during either outage.
Phase 5 -- Cost Discipline Across All Three Stages
You have three budgets ($1k, $3k, $8k) and three very different traffic levels. Design what to buy per stage, what to defer, and identify the biggest cost optimization at Stage 3.
You have three budgets ($1k, $3k, $8k) and three very different traffic levels. Design what to buy per stage, what to defer, and identify the biggest cost optimization at Stage 3.
At Stage 1 ($1k/mo), what is the most expensive premature optimization?
show answerhideAdding Redis at Stage 1 costs $300-600/mo for almost no benefit -- Postgres serves 1,800 rps at 2% utilization. The Stage 2 Redis purchase deferred saves $600/mo for 2 months = $1,200 saved before it's needed. Building the fan-out worker at Stage 1 is a full engineering quarter on infrastructure that isn't needed for 6 months.
What must be designed right at Stage 1 that cannot be changed cheaply later?
show answerhideThe schema and indexes. Adding the (followee_id) index to a 2B-row table at Stage 2 requires a full table scan -- hours of locking in production. Adding it at Stage 1 on 10M rows takes seconds. The pagination cursor format (opaque cursor vs. page number) is a client contract -- changing it at Stage 2 breaks all existing mobile clients. The media upload flow (pre-signed URL to object storage) -- migrating binary data later means touching every stored photo.
At Stage 3 ($8k/mo), what is the largest cost lever?
show answerhideCDN hit rate for photo thumbnails. At 36k reads/rps with 80KB average thumbnail: 36,000 x 80KB x $0.085/GB CDN egress = ~$9,000/mo IF all requests are cache misses. At 99% CDN hit rate: $90/mo egress. Every 1% improvement in CDN hit rate saves ~$90/mo. Use aggressive Cache-Control headers (max-age=31536000 for immutable photos) and set a thumbnail CDN cache hit rate alert as a cost KPI.
Deliverable
A per-stage budget breakdown in your defense: what you buy, what you defer, and the single largest cost optimization made. Must be within each budget.
Ready to test your design?
Open the canvas, place your components, and run the failure scenarios to get graded.