Design a URL shortener.

Overview

A URL shortener maps a long URL to a short code (dev.zy/aZ3k9Qx) and redirects anyone who opens it. The idea is simple. Doing it at internet scale is not: the service sits on the critical path of every click, so it has to stay fast, always available and hard to abuse, even when a single link goes viral.

This teardown walks through a design that handles billions of links and tens of thousands of redirects per second, and shows what happens when things go wrong.

Requirements

Out of scope

Billing, team accounts, link previews and a full analytics dashboard UI. The design leaves room for them but does not depend on them.

Capacity estimation

Back-of-the-envelope numbers decide the architecture. The read/write split is what matters most here.

Nothing here is exotic: the working set fits in a handful of Redis nodes and the data fits in a small sharded store. The hard part is the latency and availability of the read path, not the volume.

API design

A small REST surface. Creation calls accept an Idempotency-Key header so a client retry never creates a second link.

High-level architecture

Components

• CDN / Edge: terminates TLS, absorbs viral spikes by caching redirects briefly, and runs the WAF and DDoS protection. Edge logs also feed analytics, so cache hits are still counted.
• API Gateway / Load balancer: authentication for the management API, per-key and per-IP rate limiting, and routing of GET and POST traffic to separate pools.
• Redirect Service: stateless and autoscaled on requests per second. It does one thing: resolve a code to a URL as fast as possible.
• Shorten / Manage Service: validates URLs, generates codes, writes records, and handles alias, expiry and delete. It scales independently of redirects.
• Redis Cluster: caches code to URL with an LRU policy and a TTL. It also stores negative results so unknown codes do not hit the database.
• Sharded KV store: the source of truth (DynamoDB or Cassandra), partitioned by a hash of the code, replicated three ways.
• Range Allocator: hands out blocks of one million IDs to Shorten instances (details below).
• Kafka, stream job, ClickHouse: click events flow asynchronously into an analytics store so reporting never slows a redirect.

The redirect path

1. The browser requests /aZ3k9Qx. The CDN answers from its edge cache when it can (short TTL, 30 to 60 seconds).
2. On a miss the request reaches a Redirect Service instance, which checks Redis first.
3. On a Redis miss it reads the key-value store, then backfills Redis with a TTL capped at the link's expiry.
4. It publishes a click event to Kafka without waiting for an acknowledgement, then returns the redirect.

Generating short codes

The code generator has to produce unique, short and hard-to-guess codes without becoming a bottleneck. Options considered:

How the chosen approach works

1. Each Shorten instance leases a block of 1,000,000 IDs from the Range Allocator (an atomic compare-and-swap on one row in etcd or a database) and renews at 20% remaining.
2. For each request it takes the next local ID, so no network call is needed to pick a code.
3. A keyed Feistel permutation scrambles the ID into another number in the same range, so consecutive links look unrelated.
4. The result is Base62-encoded into 7 characters and written with a conditional put (if_not_exists).

A crashed instance wastes up to one million unused IDs. With 3.5 trillion possible codes and only ~6 billion used in five years, that waste is irrelevant. Region IDs are placed in the high bits of the ID, so two regions can never hand out the same code.

Custom aliases skip the generator: they are written with the same conditional put, and a conflict returns 409. Aliases are normalised, checked against a reserved-word list (api, admin, login) and length-limited.

Data model and storage

The dominant access pattern is a single-key lookup by code with no joins, which fits a key-value or wide-column store. Account and billing data lives separately in a relational database.

• Sharding: partition by hash of the code, so writes and reads spread evenly. A managed store handles rebalancing.
• Replication: three replicas per partition, quorum writes, with cross-region replication for disaster recovery.
• Expiry: the store's native TTL removes expired rows. The Redirect Service still checks expires_at, so a link stops working on time even before the row is physically deleted.

Caching strategy

Because reads are 99% of traffic, caching is the design. There are three layers, each with a clear job:

• Cache-aside with single-flight: when many requests miss on the same code at once, only one goes to the database and the rest wait for its result.
• TTL jitter stops many keys from expiring at the same moment.
• Negative caching (30 s) stops scanners and typos from hammering the database with unknown codes.
• Invalidation: disabling or deleting a link publishes an invalidation event to every Redis and calls the CDN purge API. The short edge TTL is the backstop.

Scaling and multi-region

• Stateless tiers scale horizontally behind the load balancer, with autoscaling on requests per second and p99 latency.
• Latency-based DNS / anycast sends each user to the nearest healthy region, and health checks remove a failed region within seconds.
• Read-your-write across regions: a new link may not have replicated yet when someone in another region clicks it. On a local miss, the service does a quorum read against a peer region before returning 404, then negative-caches the answer for a short time.
• Capacity headroom: each region is sized to absorb the traffic of a failed neighbour (N+1), so a regional outage is a capacity event, not an outage.

Analytics pipeline

Click analytics must never slow or break a redirect, so it is decoupled completely.

• The Redirect Service emits a small event (code, timestamp, hashed and truncated IP, user agent, referrer) to Kafka using a non-blocking producer with a bounded local buffer. If Kafka is unavailable, events are dropped rather than delaying the redirect.
• CDN access logs flow into the same topic, so edge-cache hits are counted too.
• A stream job deduplicates by event_id, filters bots by user-agent and IP heuristics, and aggregates into ClickHouse (per code, per minute).
• Stats endpoints read the aggregates, so dashboards cost nothing on the hot path.

The counts are at-least-once, then deduplicated: accurate enough for reporting, and explicitly not a billing-grade ledger.

Handling failures and edge cases

A design is only as good as its behaviour when things break. These are the situations it is built for:

Security and abuse prevention

Shorteners are a favourite tool for phishing and malware, so abuse controls are part of the core design, not an afterthought.

• Malicious targets: check every submitted URL against a reputation service (for example Google Safe Browsing) at creation, and re-scan periodically. Bad links are set to blocked and show a warning page instead of redirecting.
• Enumeration resistance: keyed permutation makes codes non-sequential; 404 responses are uniform and rate-limited.
• Rate limits on creation per API key and per IP; reserved words for aliases; HTTPS and HSTS everywhere.
• Privacy: store only a truncated, hashed IP; support deleting all data for an owner on request.
• Scanner isolation: any service that fetches a target URL runs in a sandbox with restricted egress, to prevent server-side request forgery.

Observability and SLOs

• SLOs: redirect availability 99.99% (about 52 minutes of downtime a year), redirect p99 under 50 ms at origin, create availability 99.9%.
• Golden signals per service: request rate, error rate, latency percentiles, saturation.
• Cache health: hit ratio per layer, eviction rate, key-value store read latency and throttling.
• Allocator health: remaining IDs per instance and lease latency, alerting well before exhaustion.
• Pipeline health: Kafka consumer lag and dropped-event counts.
• Synthetic probes create and resolve a link from several regions every minute, and page on failure.
• Safe rollouts: canary deploys, feature flags, and rollback on SLO burn.

Trade-offs and alternatives

Possible next steps

• Bloom filters per region to reject unknown codes without a database read.
• Per-link QR codes and deep links for mobile apps.
• Team workspaces, custom domains and branded short links.
• A read-through edge worker so redirects resolve entirely at the CDN.