Design a notification service.

Overview

A notification service is the single place where every product team sends email, SMS, push and in-app messages. Centralising it avoids each team re-implementing provider integrations, retries and opt-outs, and it lets us enforce user preferences and rate limits in one place.

The hard parts are not sending a message. They are never losing an accepted notification, never sending the same one twice, keeping urgent messages (like login codes) fast while a million-user campaign runs, and surviving provider outages.

Requirements

Out of scope

Building our own SMTP server or carrier connections, and the marketing UI for designing campaigns. We integrate with providers rather than replace them.

Capacity estimation

Assume 100M monthly active users and about 50M notifications a day across all channels.

Our own infrastructure scales linearly. External providers are the true limit, so the design throttles per provider and never assumes unlimited downstream capacity.

API design

The API answers 202, not 200: it promises the request is durably stored and will be processed, not that the message has been delivered. Delivery outcomes come from status queries or webhooks.

High-level architecture

Components

• Notification API: authenticates the caller, validates the payload, checks the idempotency key, applies per-tenant rate limits, and writes the request plus an outbox row in one transaction (in a transactional store such as PostgreSQL, or DynamoDB transactions). A relay then publishes to Kafka, so a crash between "saved" and "queued" cannot lose a request.
• Kafka priority lanes: separate topics and consumer pools for critical, transactional and bulk. A marketing blast can never delay a login code.
• Orchestrator: decides whether, what and where to send: dedupe, opt-outs and frequency caps, quiet hours, template rendering, and channel selection with fallback.
• Per-channel queues and workers: each channel scales and fails independently. Workers use a common provider-adapter interface, with circuit breakers and a secondary provider.
• Delivery status service: consumes worker events and provider receipts and maintains the notification state machine.
• Retry topics and DLQ: failures are retried with backoff, and permanently failing messages are parked for inspection.

Life of a notification

1. The producer sends the request with an idempotency key. The API stores it and replies 202 Accepted.
2. The outbox relay publishes it to the right priority lane.
3. The orchestrator consumes it, checks the dedupe cache and the user's preferences, renders the template in the user's locale, and picks channels.
4. The message goes to the channel queue. A worker pulls it, respects the provider rate limit, and calls the provider.
5. The worker reports SENT, RETRY or FAILED. Later, the provider's webhook reports DELIVERED or BOUNCED.
6. The status service updates the record and, if requested, calls the producer's webhook.

Delivery tracking and retries

Retry policy

• Classify errors: retryable (5xx, timeouts, 429) versus permanent (invalid address, unregistered device token, hard bounce).
• Retry retryable errors with exponential backoff and jitter (for example 1 s, 5 s, 30 s, 5 min), through retry topics so a slow message never blocks the main lane.
• Every notification has a TTL. A login code that could not be sent within five minutes is dropped, because a late code is worse than none.
• After the maximum attempts, or on a permanent error, the message moves to DEAD_LETTER or BOUNCED. Permanent failures also update the suppression list and deactivate bad device tokens, so we stop trying.

Delivery guarantees and idempotency

True exactly-once delivery across an external provider is impossible: if a worker crashes after the provider accepts a message but before we record it, we cannot know. The design therefore aims for at-least-once processing with idempotency at every step, which users experience as exactly-once.

• At the API: the idempotency key (kept for 24 hours) returns the original notification ID on a retry.
• At the orchestrator: a dedupe key (tenant + notification id, plus a content hash for category-level dedupe) stops reprocessing after a crash or redelivery. Kafka offsets commit only after processing succeeds.
• At the worker: the provider's own idempotency token is used where it exists (many email and push APIs support one). The remaining duplicate window is tiny and accepted; for critical messages, content is naturally safe to repeat (an OTP resend).
• At the status service: each transition carries a version, and stale or duplicate events are ignored.

Priority, fan-out and campaigns

• Isolation by lane: critical, transactional and bulk have separate topics, consumer groups and provider quotas. Under pressure, bulk waits; critical does not.
• Streaming fan-out: the audience resolver pages through user IDs, and the chunker publishes small batches, so a 10M-user campaign uses constant memory and can be paused mid-flight.
• Throttling: workers use a token bucket per provider account and tenant, adapting downward on 429 responses and honouring Retry-After.
• Frequency caps and quiet hours: marketing messages respect per-user daily limits and are deferred to the user's local window (stored as an IANA time zone, so daylight saving is handled).
• Scheduling: future sends sit in a time-bucketed store (for example a Redis sorted set or DynamoDB with TTL) and a scheduler releases them into the right lane when due.

Channels and providers

Each provider sits behind the same adapter interface (send, parse_receipt, classify_error). Adding or replacing a provider does not touch the orchestrator. A circuit breaker per provider opens on rising error rates and switches traffic to the secondary.

Data model and storage

• Intake requests and the outbox live in a transactional store (PostgreSQL, or DynamoDB with transactions) so the request and its outbox row commit atomically; processed rows are archived after a short retention.
• Notification history and deliveries are write-heavy and time-ordered, so they live in a wide-column store (Cassandra or DynamoDB) partitioned by user. The most recent 90 days stay hot; older records move to object storage as Parquet.
• Preferences and devices are read on every send, so they are cached in Redis with event-driven invalidation and backed by a relational database or DynamoDB.
• Payloads may contain personal data: they are encrypted at rest, redacted from logs, and covered by retention rules.

Handling failures and edge cases

Most of the real work in a notification system is what happens when something goes wrong:

Security and compliance

• Authentication and tenancy: API keys or OAuth per tenant, quotas per tenant, and strict isolation of data and rate limits.
• Consent and opt-out: unsubscribe links and headers, a global suppression list, and recorded consent for marketing (GDPR, CAN-SPAM, TCPA, local DND rules).
• Webhook security: verify provider signatures; sign our outgoing webhooks; replay-protect with timestamps.
• Abuse controls: block use of the service to spam; template variable escaping to prevent injection into email and HTML.
• Data protection: encrypt payloads, minimise stored personal data, redact logs, and support deletion requests.
• Auditability: an immutable audit trail of who sent what to whom and when.

Observability and SLOs

• SLOs: ingestion availability 99.99% with p99 under 100 ms; critical lane from accepted to provider hand-off p99 under 5 s; transactional under 60 s; bulk within the campaign window.
• Lane health: consumer lag and "time in queue" per lane, the first signal that something is backing up.
• Provider health: success rate, latency, throttling and bounce rate per provider and country, driving circuit breakers and alerts.
• Delivery funnel: accepted, sent, delivered, opened, per channel and template.
• Tracing: one notification_id follows a message across every service and log line.
• Synthetic canaries: a test notification on each channel and provider every minute, alerting on missing delivery.

Trade-offs and alternatives

Possible next steps

• Per-user send-time optimisation and smart digests that batch low-priority messages.
• A/B testing of templates and channels, with delivery and engagement analytics.
• Channel fallback rules (push, then SMS, then email) for critical messages.
• Additional channels such as WhatsApp and voice through the same adapter interface.