Design a URL Shortener: Step-by-Step System Design

The **URL Shortener** (such as TinyURL, bit.ly, or ow.ly) is the absolute classic question in system design interviews. While the core concept appears deceptively simple—take a long URL and return a short alias—designing a system that handles hundreds of millions of redirections with ultra-low latency, high availability, and massive write throughput requires deep understanding of distributed architectures, caching layers, and database partitioning.

This comprehensive, step-by-step guide breaks down how to design a production-grade URL shortener from scratch, covering requirements estimation, data modeling, encoding algorithms, database strategies, and scaling patterns.

Phase 1: Requirements Estimation and Capacity Planning

Before drawing architectural diagrams, you must establish functional and non-functional requirements alongside back-of-the-envelope estimations.

• Functional Requirements: Given a long URL, the system generates a unique shorter alias. When a user visits the short link, they are HTTP redirected (301/302) to the original URL. Users should also be able to specify custom short links.
• Non-Functional Requirements: Highly available, ultra-low latency redirection (< 10ms), short links must be non-guessable and secure.
• Traffic Estimates: Assume 100 million new URL shortenings per month (Writes) and a 10:1 Read-to-Write ratio, equaling 1 billion redirects per month (Reads).

# Back-of-the-envelope calculations:
# Write QPS = 100,000,000 / (30 days * 24 hours * 3600 seconds) ≈ 38.5 requests/sec
# Read QPS = 1,000,000,000 / (30 days * 24 hours * 3600 seconds) ≈ 385 requests/sec
# Storage for 5 years = 100M/month * 60 months = 6 billion URLs
# Assuming each record is 500 bytes: 6B * 500 bytes = 3 TB of total storage over 5 years.

Phase 2: API Endpoints Design

Define clean RESTful endpoints for generating short URLs and handling redirections.

# 1. Create Short URL Endpoint
POST /api/v1/data/shorten
Request Body:
{
    "long_url": "https://www.codecrown.in/devops/tutorials/design-url-shortener",
    "custom_alias": "my-url-guide"
}
Response (201 Created):
{
    "short_url": "https://cc.in/my-url-guide",
    "expires_at": "2031-10-18T00:00:00Z"
}

# 2. Redirection Endpoint
GET /{short_alias}
Response (301 Moved Permanently or 302 Found):
Headers: Location: https://www.codecrown.in/devops/tutorials/design-url-shortener

Phase 3: Database Design and Data Modeling

Because the access pattern is read-heavy with simple key-value lookups (`short_hash` -> `long_url`), a relational database (PostgreSQL) or NoSQL key-value store (Cassandra, DynamoDB) works effectively.

-- PostgreSQL URL Table Schema
CREATE TABLE urls (
    id BIGSERIAL PRIMARY KEY,
    short_hash VARCHAR(10) UNIQUE NOT NULL,
    long_url TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    user_id BIGINT
);

-- Create index for lightning-fast hash lookups
CREATE INDEX idx_short_hash ON urls(short_hash);

Phase 4: Hash Generation and Base62 Encoding

How do we generate a compact 6 to 8 character short hash for long URLs? There are two primary approaches:

• Approach 1: MD5 / SHA-256 Hashing: Hashing the long URL produces a 128/256-bit string. Taking the first 6-8 characters can lead to hash collisions. If a collision occurs, append a salt string and rehash.
• Approach 2: Base62 Conversion (Recommended): Use an auto-incrementing database ID (e.g., `2000000000`) and convert it into a Base62 string using characters `[a-z, A-Z, 0-9]`. Base62 ensures zero collisions and perfectly compact short aliases.

// Python Base62 encoder snippet
alphabet = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"

def base62_encode(num):
    if num == 0:
        return alphabet[0]
    arr = []
    while num:
        num, rem = divmod(num, 62)
        arr.append(alphabet[rem])
    return ''.join(reversed(arr))

Phase 5: Caching and High-Availability Scaling

To achieve sub-10ms latency under 1,000+ Read QPS, caching is mandatory.

• Redis Caching Layer: Place Redis in front of the database. When a redirection request arrives, check Redis first. If a cache miss occurs, query the PostgreSQL database, populate Redis, and return the result.
• Cache Eviction Policy: Use Least Recently Used (LRU) eviction so that viral, frequently accessed short links remain in memory.
• Load Balancing: Deploy multiple stateless web servers behind an Nginx or AWS Application Load Balancer (ALB) to distribute incoming traffic smoothly.

Conclusion

Designing a URL shortener requires balancing storage efficiency, collision-free hash generation via Base62 encoding, and low-latency read performance through Redis caching. Mastering these distributed systems principles prepares engineers to tackle complex enterprise scaling challenges.