Notification System

Problem Statement & Requirements

Functional Requirements

• Support multiple notification channels: push (mobile), SMS, email, in-app (web/mobile badge + feed)
• Producers (any internal service) can trigger notifications via an API
• Template-based content with variable substitution (e.g., "{{user}} liked your photo")
• User notification preferences — per-channel opt-in/out, quiet hours, frequency caps
• Notification feed — paginated list of past notifications (in-app inbox)
• Delivery tracking — sent, delivered, opened/clicked, failed
• Priority levels — critical (2FA codes, payment alerts) vs. marketing (promos)
• Rate limiting — prevent notification fatigue (max N notifications per user per hour)
• Scheduling — send at a future time, timezone-aware
• Batching / digest — group related notifications ("3 people liked your photo" instead of 3 separate)

Non-Functional Requirements

• High throughput — 1B+ notifications/day
• Low latency for critical — 2FA codes delivered within 5 seconds
• At-least-once delivery — no notification silently dropped
• Exactly-once display — dedup on the client side
• Scalability — handle viral events (celebrity post → millions of notifications in seconds)
• Observability — per-channel delivery rate, failure rate, open rate

Out of Scope

• Content generation (the calling service provides the content)
• User-facing notification creation (this is an infrastructure service)
• Rich media rendering (handled by client apps)

Scale Estimations

Traffic

Fanout Scenarios

Storage

Third-Party Costs

High-Level Architecture

Detailed Component Architecture

API Design

Send Notification

POST /api/v1/notifications
Content-Type: application/json
X-Idempotency-Key: "order-123-shipped"

Request:
{
  "recipients": {
    "user_ids": ["u123", "u456"],          // explicit user list
    // OR
    "segment": "premium_users",            // audience segment
    // OR
    "topic": "breaking_news"               // pub/sub topic
  },
  "template_id": "order_shipped",
  "template_data": {
    "order_id": "ORD-789",
    "item_name": "Wireless Headphones",
    "tracking_url": "https://track.example.com/xyz"
  },
  "channels": ["push", "email"],           // override user prefs (optional)
  "priority": "high",                      // critical | high | normal | low
  "schedule_at": "2026-04-04T09:00:00Z",   // future send (optional)
  "ttl": 3600,                              // expire if not delivered in 1h
  "collapse_key": "order_update_789",       // replace previous with same key
  "metadata": {
    "source": "order-service",
    "campaign_id": "spring_sale_2026"
  }
}

Response (202 Accepted):
{
  "notification_id": "n_abc123",
  "status": "queued",
  "estimated_recipients": 2,
  "channels": ["push", "email"]
}

Query Notification Feed (In-App)

GET /api/v1/users/{user_id}/notifications?cursor=abc&limit=20

Response:
{
  "notifications": [
    {
      "id": "n_abc123",
      "type": "order_shipped",
      "title": "Your order has shipped!",
      "body": "Wireless Headphones is on the way.",
      "image_url": "https://...",
      "action_url": "https://track.example.com/xyz",
      "read": false,
      "created_at": "2026-04-03T14:30:00Z"
    },
    ...
  ],
  "next_cursor": "def",
  "unread_count": 7
}

User Preferences

PUT /api/v1/users/{user_id}/notification-preferences

{
  "channels": {
    "push": { "enabled": true },
    "email": { "enabled": true, "digest": "daily" },
    "sms": { "enabled": false }
  },
  "quiet_hours": {
    "enabled": true,
    "start": "22:00",
    "end": "08:00",
    "timezone": "America/New_York"
  },
  "categories": {
    "marketing": { "enabled": false },
    "social": { "enabled": true, "channels": ["push"] },
    "transactional": { "enabled": true, "channels": ["push", "email", "sms"] }
  }
}

Data Model

Notification Log (Cassandra)

Delivery Status (ClickHouse — Analytics)

Fanout Strategies

The Fanout Problem

Strategy 1: Fanout-on-Write (Push Model)

Strategy 2: Fanout-on-Read (Pull Model)

Strategy 3: Hybrid (Recommended)

Notification Processing Pipeline

Full Flow Diagram

Priority Routing

Notification Deduplication & Idempotency

Push Notification Deep Dive

Push Architecture (iOS + Android + Web)

APNs (Apple Push Notification Service) Integration

FCM (Firebase Cloud Messaging) Integration

Notification Batching & Digests

Rate Limiting & Quiet Hours

Delivery Tracking & Analytics

Template Engine

Scheduled & Timezone-Aware Notifications

Workshop note · added for the website’s common reading format

Walk through three failure moments in Notification System: a request times out before its result is known; a dependency becomes slow rather than unavailable; and a process restarts after committing state but before acknowledging it.

For each case, name the authoritative record, define a safe retry, cap resource usage, and describe what the caller sees. Retry + DLQ should help detect and contain the problem: Retry transient failures with backoff and jitter; quarantine persistent failures for controlled replay.

Tradeoffs & Design Decisions Summary

Retry & Dead Letter Queue

Production Architecture

Workshop note · added for the website’s common reading format

Rebuild Notification System from memory, then change one assumption: ten times more traffic, a new region, or a stricter consistency requirement. Which component must change first—and which does not?

Compare Policy engine with the same boundary in a related design. Write down one alternative you rejected, what it would simplify, and when you would choose it instead. Follow the source link at the end of this article to explore the original document.

Interview Tips

1. Start by clarifying channels — "Which channels do we need? Push, email, SMS, in-app? All of them?" This scopes the design.

2. Draw the pipeline first — Producer → API → Kafka → Workers → Providers. This is the backbone.

3. Separate critical from non-critical immediately — "2FA codes and marketing promos must not share the same queue." This shows operational maturity.

4. Discuss fanout for broadcast — "What happens when a celebrity posts and 50M users need a notification?" Show chunked fanout, FCM topics, and background workers.

5. User preferences are a first-class concern — Don't treat them as an afterthought. "Users must be able to opt out per channel, set quiet hours, and control frequency."

6. Rate limiting prevents fatigue — "Without rate limiting, a burst of social activity could send 50 push notifications in a minute." Show per-user per-channel limits.

7. Idempotency is critical — "The producer might retry. Kafka gives at-least-once. We need dedup at every layer." Show the idempotency key pattern.

8. Don't forget device token management — "Tokens go stale when users uninstall the app. APNs returns 410 → we must clean up."

9. Batching shows design depth — "3 people liked your photo" instead of 3 separate notifications. Explain time-window and count-threshold strategies.

10. End with observability — "How do we know it's working? Delivery rate dashboards, p99 latency, DLQ depth alerts." This is what separates a design from a production system.