Requirements & scope
Problem Statement & Requirements
Functional Requirements
- Upload photos -- users upload images with caption, tags, location, and filters
- News feed -- see a personalized feed of photos from followed users
- Follow/unfollow -- follow other users to see their content
- Like & comment -- interact with posts
- Explore / discover -- browse trending and recommended content
- Stories -- ephemeral 24-hour photo/video content (brief mention)
- User profiles -- view user's posts grid, follower/following counts, bio
- Search -- search by username, hashtag, or location
- Notifications -- new followers, likes, comments, mentions
Non-Functional Requirements
- High availability -- 99.99% uptime
- Low latency -- feed loads in < 500 ms, image loads in < 200 ms
- Consistency -- eventual consistency acceptable for feed; strong for follow/unfollow
- Read-heavy -- read:write ratio ~100:1
- Scalability -- 500M+ DAU, 100M+ photos uploaded/day
- Durability -- uploaded photos never lost
Out of Scope
- Reels / short-form video (covered in video streaming design)
- Messaging / DMs (covered in chat system design)
- Ads / monetization platform
- Shopping / e-commerce integration
Scale estimations
Scale Estimations
Users & Content
| Metric | Value |
|---|---|
| Total users | 2B |
| Daily Active Users (DAU) | 500M |
| Photos uploaded / day | 100M |
| Average photo size (original) | 3 MB |
| Average photo size (after processing) | 500 KB (multiple sizes) |
| Average followers per user | 200 |
| Average following per user | 200 |
| Celebrity/influencer followers | 1M - 500M |
Storage
| Metric | Value |
|---|---|
| Raw photo storage / day | 100M x 3 MB = 300 TB/day |
| Processed photos / day (4 sizes) | 100M x 4 x 200 KB = 80 TB/day |
| Total photo storage / year | ~140 PB/year |
| Metadata per photo | ~1 KB |
| Metadata storage / day | 100M x 1 KB = 100 GB/day |
Traffic
| Metric | Value |
|---|---|
| Feed requests / day | 500M DAU x 10 opens/day = 5B |
| Feed requests / sec | ~58K RPS |
| Peak feed requests / sec | ~175K RPS |
| Photo views / day | 5B feeds x 10 photos/feed = 50B |
| Photo uploads / sec | 100M / 86400 = ~1,160/sec |
Bandwidth
| Metric | Value |
|---|---|
| Feed image egress | 50B views x 200 KB = ~10 PB/day |
| Upload ingress | 300 TB/day |
| CDN offloads 95%+ of egress | Origin serves < 500 TB/day |
Layered architecture
Phase 1 of 4 · Client & edge
Start with the entry points
First, define who uses the system and where requests enter. This is the boundary of the design, not yet a complete request path.
- Mobile client
- Separate upload progress from publishing a post that other people can view.
- API gateway
- Authenticate users and route media operations separately from feed reads.
This first layer defines the system boundary. The next chapters add application logic and data dependencies. After examining the request paths and tradeoffs, we will add operational safeguards and bring the complete architecture together.
API & contracts
Phase 2 of 4 · Application & compute
Add the application logic
Keep the entry points and add the services that do the work. Give each service a clear responsibility before defining its contracts. Storage and background dependencies come next.
- Post service
- Create the post record and coordinate the asynchronous image-processing lifecycle.
- Image workers
- Generate thumbnails and variants, and validate content before publishing it widely.
- Fanout service
- Push to ordinary followers and pull celebrity content at read time to limit write amplification.
An upload session should identify the owner, expected content constraints, and upload state. Publishing a post references completed media rather than accepting an arbitrary storage URL from a client. Keep the media-processing status explicit so the app can show a pending image without accidentally publishing an inaccessible or unvalidated object.
Data model
Phase 3 of 4 · Data & async work
Connect data and background work
The services now need their dependencies: durable state, fast lookups, or work that continues outside a request. Separate these responsibilities rather than treating every component as a database.
- Blob store + CDN
- Store image bytes in object storage and deliver immutable variants through the CDN.
- Feed cache
- Cache ordered post IDs rather than duplicating large media payloads into every feed.
- Social graph
- Find intended recipients and enforce visibility rules when generating or reading feeds.
Data Model
Posts Table (MySQL / Vitess)
Social Graph (Follows)
Likes & Comments
Core design decisions
News Feed -- The Core Design Challenge
Fanout-on-Write vs Fanout-on-Read
This is THE critical design decision for Instagram/Twitter-like systems.
Hybrid Approach (Instagram's Actual Design)
Request flows
Photo Upload & Processing Flow
Image Processing Pipeline
Feed Generation Flow (Detailed)
Feed Pagination (Cursor-Based)
Performance & caching
Feed Cache Design (Redis)
Feed Ranking (Beyond Reverse Chronological)
Advanced design
Blob Storage & CDN
Fanout Service Detail
Explore / Discover Page
Stories (24-Hour Ephemeral Content)
Search
Edge cases
The image upload can finish while post creation fails. Keep enough upload-session state to retry linking the media, and clean up abandoned objects after a safe retention window. Privacy changes need to reach derived image URLs and feed caches, not just the row storing the caption.
Tradeoffs
Tradeoffs & Design Decisions Summary
| Decision | Option A | Option B | Chosen | Why |
|---|---|---|---|---|
| Feed strategy | Fanout-on-write | Fanout-on-read | Hybrid | Push for normal users (fast reads), pull for celebrities (avoid 100M writes) |
| Feed cache | Database | Redis sorted sets | Redis | O(1) reads, sorted by score, TTL for inactive users |
| Feed order | Chronological | ML-ranked | ML-ranked | Increases engagement 40%+; chronological misses relevant posts |
| Image storage | Own HDFS | S3 | S3 | Managed, infinite scale, cheap, 11 nines durability |
| Image upload | Through API server | Direct to S3 (presigned) | Direct S3 | API servers don't handle multi-MB uploads; offload to S3 |
| Image sizes | One size | 5 sizes (thumb to original) | 5 sizes | Serve appropriate size per context; saves bandwidth |
| Social graph | Graph DB (Neo4j) | MySQL + cache (TAO) | MySQL + TAO | Simpler operations, TAO cache gives graph-like perf, battle-tested at Facebook scale |
| Pagination | Offset-based | Cursor-based | Cursor | Stable under insertions/deletions; no deep-scan cost |
| Like counts | Sync DB update | Async (Kafka -> Redis -> DB) | Async | Can't do 100K writes/sec to single row; batch aggregation |
| Post metadata DB | NoSQL | MySQL (Vitess) | MySQL/Vitess | Need joins (user+post), Vitess scales MySQL horizontally |
| CDN | Single-tier | Multi-tier (edge + regional + origin shield) | Multi-tier | 95%+ cache hit at edge; origin shield collapses duplicate origin fetches |
| Stories storage | Same as posts | Separate with TTL | Separate + TTL | Auto-expire after 24h; different access patterns from permanent posts |
Reliability & fault tolerance
Phase 4 of 4 · Operations & resilience
Add operational safeguards
With the request paths and tradeoffs established, add the controls that help detect trouble and keep the system operating. The final diagram retains every layer introduced so far.
- Telemetry
- Watch processing backlog, feed staleness, and image delivery failures separately.
Store accepted originals durably and make derivative generation retryable. A failed thumbnail worker should leave a recoverable job, not a permanently broken post. Track processing failures separately from upload failures, and ensure deleting a post eventually removes its derivatives without racing an in-progress resize job.
Production architecture
High-Level Architecture
Detailed Architecture
Production Architecture
Further exploration
Change one assumption: a photo can move from public to private after it has gone viral. Trace every copy and pointer—object storage, signed URLs, CDN entries, feeds, and notifications. Decide which guarantees are achievable immediately and which require a clearly bounded expiry window.
Interview playbook
Interview Tips
Start with the feed -- "The core challenge of Instagram is generating a personalized feed for 500M daily users. Let me design the feed system first." This shows you know what matters.
Fanout-on-write vs fanout-on-read is THE question -- Draw both approaches. Explain the celebrity problem. Show the hybrid solution. This is what interviewers want to hear.
Feed cache numbers -- "500M users x 500 posts x 30 bytes = 7.5 TB in Redis. With eviction of inactive users, ~2 TB. That's ~30-40 Redis nodes." Shows you can do the math.
Image processing is important but brief -- "Upload directly to S3 via presigned URL. Process async: resize to 5 versions. Store all in S3. Serve via CDN." That's 3 sentences and covers it.
Don't forget the social graph -- "The follows table has 100B rows. We shard by follower_id for 'who do I follow' queries and maintain a reverse index for 'who follows me' for fanout."
CDN is critical -- "50B image views/day at 200 KB each = 10 PB/day. That's CDN territory. 95%+ cache hit rate because images are immutable."
Cursor pagination, not offset -- If feed pagination comes up, explain why cursor-based pagination is stable under concurrent writes. This is a detail that shows real system experience.
Ranked feed shows ML awareness -- "Instagram's feed isn't chronological. We rank by interest, recency, relationship, and engagement velocity. Two-stage pipeline: 500 candidates from cache, then ML ranking."
Scale numbers anchor the design -- "100M uploads/day = 1,160/sec. 5B feed reads/day = 58K RPS. Read:write is 50:1. This is a read-heavy system, which is why we pre-compute feeds."
Stories are a nice add-on -- "Stories are ephemeral posts with a 24-hour TTL. Redis with auto-expire. Separate from the main feed. Show as horizontal tray sorted by recency." Quick, clean, shows breadth.