Notification System

Problem Statement & Requirements

Functional Requirements

• Support multiple notification channels: push (mobile), SMS, email, in-app (web/mobile badge + feed)
• Producers (any internal service) can trigger notifications via an API
• Template-based content with variable substitution (e.g., "{{user}} liked your photo")
• User notification preferences — per-channel opt-in/out, quiet hours, frequency caps
• Notification feed — paginated list of past notifications (in-app inbox)
• Delivery tracking — sent, delivered, opened/clicked, failed
• Priority levels — critical (2FA codes, payment alerts) vs. marketing (promos)
• Rate limiting — prevent notification fatigue (max N notifications per user per hour)
• Scheduling — send at a future time, timezone-aware
• Batching / digest — group related notifications ("3 people liked your photo" instead of 3 separate)

Non-Functional Requirements

• High throughput — 1B+ notifications/day
• Low latency for critical — 2FA codes delivered within 5 seconds
• At-least-once delivery — no notification silently dropped
• Exactly-once display — dedup on the client side
• Scalability — handle viral events (celebrity post → millions of notifications in seconds)
• Observability — per-channel delivery rate, failure rate, open rate

Out of Scope

• Content generation (the calling service provides the content)
• User-facing notification creation (this is an infrastructure service)
• Rich media rendering (handled by client apps)

Scale Estimations

Traffic

Fanout Scenarios

Storage

Third-Party Costs

This first layer defines the system boundary. The next chapters add application logic and data dependencies. After examining the request paths and tradeoffs, we will add operational safeguards and bring the complete architecture together.

API Design

Send Notification

POST /api/v1/notifications
Content-Type: application/json
X-Idempotency-Key: "order-123-shipped"

Request:
{
  "recipients": {
    "user_ids": ["u123", "u456"],          // explicit user list
    // OR
    "segment": "premium_users",            // audience segment
    // OR
    "topic": "breaking_news"               // pub/sub topic
  },
  "template_id": "order_shipped",
  "template_data": {
    "order_id": "ORD-789",
    "item_name": "Wireless Headphones",
    "tracking_url": "https://track.example.com/xyz"
  },
  "channels": ["push", "email"],           // override user prefs (optional)
  "priority": "high",                      // critical | high | normal | low
  "schedule_at": "2026-04-04T09:00:00Z",   // future send (optional)
  "ttl": 3600,                              // expire if not delivered in 1h
  "collapse_key": "order_update_789",       // replace previous with same key
  "metadata": {
    "source": "order-service",
    "campaign_id": "spring_sale_2026"
  }
}

Response (202 Accepted):
{
  "notification_id": "n_abc123",
  "status": "queued",
  "estimated_recipients": 2,
  "channels": ["push", "email"]
}

Query Notification Feed (In-App)

GET /api/v1/users/{user_id}/notifications?cursor=abc&limit=20

Response:
{
  "notifications": [
    {
      "id": "n_abc123",
      "type": "order_shipped",
      "title": "Your order has shipped!",
      "body": "Wireless Headphones is on the way.",
      "image_url": "https://...",
      "action_url": "https://track.example.com/xyz",
      "read": false,
      "created_at": "2026-04-03T14:30:00Z"
    },
    ...
  ],
  "next_cursor": "def",
  "unread_count": 7
}

User Preferences

PUT /api/v1/users/{user_id}/notification-preferences

{
  "channels": {
    "push": { "enabled": true },
    "email": { "enabled": true, "digest": "daily" },
    "sms": { "enabled": false }
  },
  "quiet_hours": {
    "enabled": true,
    "start": "22:00",
    "end": "08:00",
    "timezone": "America/New_York"
  },
  "categories": {
    "marketing": { "enabled": false },
    "social": { "enabled": true, "channels": ["push"] },
    "transactional": { "enabled": true, "channels": ["push", "email", "sms"] }
  }
}

Data Model

Notification Log (Cassandra)

Delivery Status (ClickHouse — Analytics)

Fanout Strategies

The Fanout Problem

Strategy 1: Fanout-on-Write (Push Model)

Strategy 2: Fanout-on-Read (Pull Model)

Strategy 3: Hybrid (Recommended)

Notification Processing Pipeline

Full Flow Diagram

Priority Routing

Notification Deduplication & Idempotency

Push Notification Deep Dive

Push Architecture (iOS + Android + Web)

APNs (Apple Push Notification Service) Integration

FCM (Firebase Cloud Messaging) Integration

Notification Batching & Digests

Rate Limiting & Quiet Hours

Delivery Tracking & Analytics

Template Engine

Scheduled & Timezone-Aware Notifications

A user disables promotional email after a campaign has already entered the queue. Check the relevant preference again before delivery rather than relying only on the snapshot taken when the job was created. Provider timeouts need similar care: retain the provider’s operation ID and distinguish an unknown outcome from a confirmed rejection.

Tradeoffs & Design Decisions Summary

Retry & Dead Letter Queue

High-Level Architecture

Detailed Component Architecture

Production Architecture

Send an urgent security alert while a large newsletter campaign is draining. Trace where the urgent work might still wait: queue partitions, worker capacity, and provider quotas. Separate priority queues are useful only if the downstream resources can also reserve capacity for urgent traffic.

Interview Tips

1. Start by clarifying channels — "Which channels do we need? Push, email, SMS, in-app? All of them?" This scopes the design.

2. Draw the pipeline first — Producer → API → Kafka → Workers → Providers. This is the backbone.

3. Separate critical from non-critical immediately — "2FA codes and marketing promos must not share the same queue." This shows operational maturity.

4. Discuss fanout for broadcast — "What happens when a celebrity posts and 50M users need a notification?" Show chunked fanout, FCM topics, and background workers.

5. User preferences are a first-class concern — Don't treat them as an afterthought. "Users must be able to opt out per channel, set quiet hours, and control frequency."

6. Rate limiting prevents fatigue — "Without rate limiting, a burst of social activity could send 50 push notifications in a minute." Show per-user per-channel limits.

7. Idempotency is critical — "The producer might retry. Kafka gives at-least-once. We need dedup at every layer." Show the idempotency key pattern.

8. Don't forget device token management — "Tokens go stale when users uninstall the app. APNs returns 410 → we must clean up."

9. Batching shows design depth — "3 people liked your photo" instead of 3 separate notifications. Explain time-window and count-threshold strategies.

10. End with observability — "How do we know it's working? Delivery rate dashboards, p99 latency, DLQ depth alerts." This is what separates a design from a production system.