Event-Driven Architecture

Problem Statement & Requirements

Design an event-driven architecture platform that enables loosely-coupled microservices to communicate asynchronously through events, supports event sourcing for audit/replay, provides schema governance, and handles the operational challenges of distributed event systems at scale — the architectural backbone behind systems at Netflix, Uber, LinkedIn, and Shopify.

Functional Requirements

• Event bus / broker — publish-subscribe messaging backbone (Kafka-based) with topics, partitions, and consumer groups
• Event schema registry — centralized schema management with versioning, compatibility checks, and serialization (Avro/Protobuf)
• Event catalog / discovery — searchable catalog of all event types across the organization, their schemas, owners, and consumers
• Event sourcing — store state as a sequence of immutable events; reconstruct current state by replaying events
• CQRS (Command Query Responsibility Segregation) — separate write models (commands → events) from read models (materialized views)
• Saga orchestration — coordinate multi-service transactions via event-driven sagas with compensation
• Dead letter queue (DLQ) — capture failed events for inspection, replay, and manual resolution
• Event replay — replay historical events to rebuild state, backfill new consumers, or debug issues
• Exactly-once semantics — idempotent event processing to prevent duplicate side effects
• CDC (Change Data Capture) — stream database changes as events (Debezium pattern)

Non-Functional Requirements

• High throughput — handle 5M+ events/second across the platform
• Low latency — event publish-to-consume in < 50 ms (p99)
• Durability — zero event loss once acknowledged
• Ordering — guaranteed ordering within a partition/entity
• Schema evolution — producers and consumers evolve independently without breaking each other
• Observability — trace events across services (distributed tracing), monitor consumer lag, detect broken consumers
• Multi-team governance — hundreds of teams publish/consume events; need ownership, discovery, and compatibility rules

Out of Scope

• Full message broker internals (see Distributed Message Queue design)
• Specific business domain modeling (DDD bounded contexts)
• Stream processing engine internals (Kafka Streams / Flink)

Scale Estimations

Traffic

Storage

Bandwidth

Hardware Estimate

This first layer defines the system boundary. The next chapters add application logic and data dependencies. After examining the request paths and tradeoffs, we will add operational safeguards and bring the complete architecture together.

Event Design & Schema

Event Envelope (Standard Structure)

{
  "event_id": "evt-7f3a9b2c-1234-5678-abcd-ef0123456789",
  "event_type": "orders.placed",
  "event_version": "3.1",
  "source": "order-service",
  "timestamp": "2026-04-03T10:00:00.123Z",
  "correlation_id": "req-abc123",
  "causation_id": "evt-previous-xyz",
  "partition_key": "order-789",
  "tenant_id": "acme-corp",
  "metadata": {
    "user_id": "user-alice",
    "trace_id": "trace-456",
    "span_id": "span-789"
  },
  "data": {
    "order_id": "order-789",
    "customer_id": "cust-123",
    "items": [
      {"sku": "SKU-001", "quantity": 2, "unit_price": 2999}
    ],
    "total_amount": 5998,
    "currency": "usd",
    "shipping_address": {
      "city": "New York",
      "country": "US"
    }
  }
}

Event Naming Conventions

Schema Registry & Evolution

Give every event a stable ID, type, schema version, occurrence time, and relevant business key. Consumers keep their own progress and derived records. An outbox row belongs in the same transaction as the business change it announces, so a committed order cannot disappear between the database write and publication.

An event describes a completed fact; a command asks for an action that may be refused. Keep that distinction in names and contracts. Use an outbox or equivalent atomic publication boundary, expect redelivery, and make each consumer’s side effects idempotent under a durable operation identity.

An order transaction writes the order and its outbox event together. A relay publishes the committed event, and each subscriber records or recognizes the event ID before applying its business action. Later events describe the results. A saga tracks the larger workflow and requests compensating actions when a required step cannot complete.

Partition by the key whose events must remain ordered, then scale consumers across independent partitions. Batch where the latency budget allows it, but watch consumer lag rather than producer throughput alone. A slow subscriber should not block unrelated subscribers; retention must still cover the recovery time the product promises.

Event Sourcing Deep Dive

Concept

Snapshots (Performance Optimization)

CQRS (Command Query Responsibility Segregation)

Saga Pattern — Distributed Transactions

The Problem

Choreography-Based Saga (No Central Coordinator)

Orchestration-Based Saga (Central Coordinator)

Change Data Capture (CDC)

Handling Edge Cases

Event Ordering Across Services

Problem: OrderPlaced and PaymentCompleted arrive out of order

  Event 1: OrderPlaced (published at T=0)
  Event 2: PaymentCompleted (published at T=1)

  Due to partitioning / network:
  Consumer sees: PaymentCompleted BEFORE OrderPlaced

Solution:
  1. Partition by entity ID (order_id)
     → All events for one order go to same partition
     → Within partition: strict ordering guaranteed
     ✅ Handles 99% of ordering needs

  2. For cross-entity ordering (rare):
     → Causation chain: each event carries causation_id
     → Consumer buffers events, processes in causal order
     → Or: accept eventual consistency (out-of-order OK for analytics)

  3. For strict global ordering (very rare):
     → Single partition (kills parallelism — avoid if possible)
     → Or: Lamport timestamps for causal ordering

Duplicate Events (Exactly-Once Processing)

Problem: Kafka guarantees at-least-once delivery
  → Consumer may see the same event twice (rebalance, retry)

Solutions:
  Layer 1: Idempotent Processing
    Each event has unique event_id
    Consumer tracks processed event_ids in local store
    Before processing: check "have I seen event_id X?"
    → Yes: skip (already processed)
    → No: process + record event_id

  Layer 2: Transactional Outbox (for producers)
    Instead of publishing directly to Kafka:
    1. Write event to outbox table in SAME DB transaction as state change
    2. Background poller reads outbox → publishes to Kafka → marks as published
    3. If service crashes after DB commit but before Kafka publish:
       → Poller retries → event eventually published
    4. If service crashes before DB commit:
       → Nothing in outbox → no duplicate → safe

  Layer 3: Kafka Transactions (for Kafka-to-Kafka processing)
    Read from input topic + write to output topic + commit offset = ATOMIC
    Kafka idempotent producer + transactional consumer = exactly-once

  Recommendation: Idempotent consumers (Layer 1) for all services
    Transactional outbox (Layer 2) for critical producers

Schema Breaking Change

Problem: Producer adds a required field → all consumers break

Prevention:
  1. Schema Registry enforces FULL compatibility
     → Reject schema that removes fields or adds required fields
     → Only optional field additions allowed

  2. If breaking change is truly needed:
     → Create NEW topic: orders.placed.v2
     → Run both v1 and v2 in parallel (migration period)
     → Consumers migrate to v2 at their own pace
     → After all consumers migrated → deprecate v1 topic
     → Timeline: 2-4 weeks migration window typical

  3. Event versioning in schema:
     → event_version field in envelope
     → Consumer checks version → delegates to appropriate handler
     → v1 handler and v2 handler coexist in consumer code

Consumer Falls Behind (Lag)

Problem: Analytics consumer processes at 50K events/sec
  Producer writes 100K events/sec → lag grows unboundedly

Detection:
  Monitor: consumer_lag = latest_offset - committed_offset
  Alert: lag > 1M events OR lag growing for > 10 minutes

Response:
  1. Scale out consumers (add instances to consumer group)
     → Kafka rebalances partitions across more consumers
     → Max parallelism = number of partitions

  2. If at max consumers (= partition count):
     → Optimize processing (batch DB writes, reduce per-event cost)
     → Or: add more partitions (requires topic reconfiguration)

  3. For non-critical consumers (analytics):
     → Accept lag (process at own pace, catch up during off-peak)
     → Skip old events if > 24h behind (stale data less useful)

  4. For critical consumers (payment):
     → Auto-scale aggressively
     → Alert if lag > 1 minute
     → This consumer should NEVER fall behind

Tradeoffs & Design Decisions Summary

Dead Letter Queue & Error Handling

Observability for Event-Driven Systems

High-Level Architecture

Component Breakdown

Full System Architecture (Production-Grade)

When to Use (and NOT Use) Event-Driven Architecture

Interview Tips

1. Start with WHY events — "Event-driven architecture decouples producers from consumers. The Order Service publishes OrderPlaced without knowing or caring who consumes it. Payment, Inventory, Notifications, Analytics all react independently. Adding a new consumer requires zero changes to the producer."

2. Events vs Commands is a fundamental distinction — Events: past tense, broadcast, immutable facts ("OrderPlaced"). Commands: imperative, directed, one target ("ShipOrder"). Events describe what happened; commands describe what should happen. Mixing them up creates tight coupling.

3. Schema Registry prevents breaking changes — "Every event schema is registered with FULL compatibility enforcement. Adding an optional field = safe. Removing a field = rejected by registry. Producers and consumers evolve at their own pace. This is what prevents 'I deployed and broke 10 downstream services.'"

4. Saga pattern replaces distributed transactions — "2PC locks across 4 databases = fragile and slow. Saga coordinates via events with compensating actions. Payment fails after inventory reserved? Publish InventoryReleaseRequested. Each step is undoable. Orchestration-based for complex flows, choreography for simple 2-3 step flows."

5. Event Sourcing is powerful but use selectively — Don't event-source everything. "Order management, payment transactions, audit-critical domains = event sourcing (full history, time-travel, replay). User profiles, configuration = CRUD (simpler, sufficient). Event sourcing adds complexity; use it where the audit/replay value justifies it."

6. CQRS enables optimal read models — "Write model normalized for consistency. Read models denormalized for query performance. Order dashboard in PostgreSQL, search in Elasticsearch, analytics in ClickHouse — each optimized for its query pattern. All built from the same event stream."

7. CDC bridges legacy systems — "Can't modify the legacy monolith to publish events? Debezium reads the PostgreSQL WAL and publishes every INSERT/UPDATE/DELETE as an event to Kafka. Zero code changes. This is how most companies start their event-driven migration."

8. DLQ is non-negotiable — "Events that fail processing after N retries go to a Dead Letter Queue. Never silently drop events. DLQ enables inspection ('why did this fail?'), replay ('fix the bug, replay events'), and alerting ('DLQ depth > 0 = someone needs to look')."

9. Observability is the hardest part — "A synchronous REST call chain is easy to trace. An async event flow across 5 services via Kafka is not. Propagate correlation_id + trace_id through every event. Build an event topology dashboard showing which services produce/consume which topics. Monitor consumer lag religiously."

10. End with the honest tradeoff — "Events give you loose coupling, independent scaling, and audit trails. They cost you eventual consistency, debugging complexity, and operational overhead. The right question isn't 'should we use events?' — it's 'which interactions benefit from decoupling enough to justify the complexity?'"