How to design a scalable URL shortener service like Bitly?

Question

1

Overview

Designing a URL shortener service requires building a highly scalable, low-latency web application capable of handling a high read-to-write ratio (often 100:1). The core functional requirements include generating a shortened alias for a long URL and redirecting users when they hit the short link.

Key System Requirements

High Availability: The redirection service must not go down.
Low Latency: Short link redirection should take less than 10ms.
Custom & Unique Hash Keys: Short URLs should be non-predictable and around 6-8 characters long.

Architecture & Data Flow

To support high throughput, read requests should bypass the primary SQL database and hit an in-memory caching layer such as Redis or Memcached. When a write request arrives, the app converts a unique auto-incrementing integer ID into Base62.

Base62 Encoding Implementation

Below is a Python implementation to encode unique database IDs into Base62 characters:

def encode_base62(num: int) -> str:
    chars = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
    if num == 0:
        return chars[0]
    result = []
    base = len(chars)
    while num > 0:
        result.append(chars[num % base])
        num //= base
    return "".join(reversed(result))

1 Answer

Write Your Answer

Answer 1

Introduction

Designing a scalable URL shortener like Bitly, TinyURL, or Ow.ly is a classic system design interview question. A URL shortener converts long, cumbersome web links into concise, unique aliases that redirect users to the original web address. In this article, we will break down the end-to-end architecture, capacity estimations, encoding algorithms, database choices, and caching strategies required to build a highly available system handling billions of requests.

1. Requirements Clarification

Functional Requirements

URL Shortening: Given a long URL, the service generates a shorter, unique URL (alias).
URL Redirection: When a user accesses the short URL, the service redirects them to the original long URL.
Custom Aliases: Users can optionally specify a custom short alias for their links.
Expiration: Links can optionally have an expiration date.
Analytics: Track click stats such as redirect counts, user locations, referrers, and device types.

Non-Functional Requirements

High Availability: The system must be available 99.99% of the time, as link redirection failures break user journeys.
Low Latency: Redirection must happen in real time with minimal latency (under 20ms).
Scalability: The system should seamlessly scale to support massive read traffic (100:1 read-to-write ratio).
Unpredictability: Short links should not be easily guessable to prevent security scanning.

2. Capacity Estimation and Scale

Let us assume the following scale parameters for our system:

New URLs written: 500 million per month.
Read/Write ratio: 100:1 (50 billion redirections per month).
Write QPS: ~200 writes per second.
Read QPS: ~20,000 reads per second.
Storage calculation: Assuming each URL record requires 500 bytes and we store data for 5 years: 500 million * 12 months * 5 years * 500 bytes = 15 Terabytes total storage.
Memory Cache calculation: Following the 80/20 rule, 20% of links generate 80% of traffic. Daily read volume is 1.7 billion requests. 20% of 1.7 billion * 500 bytes = 170 GB RAM needed for caching.

3. API Design

We can expose simple RESTful endpoints for URL operations:

Create Short URL

POST /api/v1/data/shorten

Request Payload:
{
    "longUrl": "https://www.example.com/articles/system-design-guide?id=98765",
    "customAlias": "sys-guide",
    "expireDate": "2030-12-31T23:59:59Z"
}

Response:
{
    "shortUrl": "https://short.url/sys-guide",
    "createdAt": "2026-03-30T10:00:00Z"
}

Redirect Short URL

GET /{shortKey}

Returns HTTP Status 301 (Permanent Redirect) or 302 (Temporary Redirect) with the Location header pointing to the long URL. Note: Use 301 to reduce server load via browser caching, or 302 if you need accurate analytics on every click.

4. Key Encoding Algorithm: Base62 vs Hashing

How do we convert an auto-incrementing ID or long URL string into a short 7-character code?

Using Base62 encoding (characters [a-z, A-Z, 0-9]), a 7-character string yields 62⁷ = ~3.5 Trillion unique combinations, which is more than enough for our scale.

Approach A: Base62 Encoding of Auto-Incrementing IDs

Convert a unique numerical ID generated by the database into a Base62 string.

class Base62Converter:
    CHARS = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
    BASE = 62

    @classmethod
    def encode(cls, num: int) -> str:
        if num == 0:
            return cls.CHARS[0]
        result = []
        while num > 0:
            num, rem = divmod(num, cls.BASE)
            result.append(cls.CHARS[rem])
        result.reverse()
        return "".join(result)

    @classmethod
    def decode(cls, short_str: str) -> int:
        num = 0
        for char in short_str:
            num = num * cls.BASE + cls.CHARS.index(char)
        return num

Approach B: Key Generation Service (KGS)

To avoid single-point database bottle-necks or predictable ID sequential vulnerabilities, a standalone Key Generation Service (KGS) pre-generates random 7-character unique strings ahead of time and stores them in two key tables: Available Keys and Used Keys. When a write request arrives, the application server fetches a pre-generated key instantly.

5. System Architecture

The core architecture consists of:

API Gateway / Load Balancers: Distribute traffic using Round Robin or Least Connections.
Application Servers: Stateless nodes that handle API requests and key encoding.
Distributed Cache (Redis / Memcached): Stores hot key-value pairs (shortKey -> longUrl) to satisfy low latency reads.
NoSQL Database (Cassandra / MongoDB): Stores mappings. NoSQL scales horizontally across multiple shards effortlessly for key-value lookups.

Conclusion

Designing a system like Bitly requires balancing high read performance with strong data persistence. By using a Key Generation Service or Base62 encoding combined with a distributed caching layer (Redis) and NoSQL persistence, the system can scale to tens of thousands of requests per second while maintaining millisecond redirection speeds.