How to design a resumable, chunked file upload system for web applications?

Question

1

Overview

Uploading large files (e.g., several gigabytes) directly through standard HTTP POST requests is prone to network timeouts and connection drops. A robust system design requires chunking files on the browser client, uploading parts concurrently, and stitching them on cloud blob storage.

Key Considerations

Resumability: Record uploaded chunk hashes in a database so failed uploads resume from the last successful chunk.
Direct Cloud Uploads: Bypass application web servers by generating presigned S3 URLs to save web tier bandwidth.
Deduplication: Hash file content to avoid storing duplicate files.

Frontend Chunking JavaScript Implementation

Below is a JavaScript snippet demonstrating client-side file slicing into distinct byte-array chunks:

async function uploadFileInChunks(file, chunkSize = 5 * 1024 * 1024) {
    const totalChunks = Math.ceil(file.size / chunkSize);
    for (let index = 0; index < totalChunks; index++) {
        const start = index * chunkSize;
        const end = Math.min(file.size, start + chunkSize);
        const chunk = file.slice(start, end);
        
        const formData = new FormData();
        formData.append('chunkIndex', index);
        formData.append('data', chunk);
        
        await fetch('/api/upload-chunk', {
            method: 'POST',
            body: formData
        });
    }
}

1 Answer

Write Your Answer

Answer 1

Uploading large files over HTTP poses significant challenges, including server timeouts, high memory usage, and dropped network connections. Designing a resumable, chunked file upload system solves these issues by breaking a file into smaller segments (chunks), uploading them independently, and allowing interrupted uploads to resume seamlessly from the last successful chunk.

Core Concepts of Chunked Uploads

A chunked file upload architecture relies on four main pillars:

File Slicing: Using modern browser APIs like Blob.prototype.slice() to split files into fixed-size chunks on the client side.
Unique File Identification: Generating a unique identifier (e.g., using file metadata or a hash like MD5/SHA-256) to track upload progress across sessions.
State Tracking: Maintaining metadata on the server to record which chunks have been received.
Chunk Merging: Stitching all individual chunk files back into the original file once all parts are uploaded.

System Workflow

1. Upload Initialization

Before sending file data, the frontend requests an upload session from the server by sending file metadata (name, size, mime type, hash). The server returns an uploadId and a list of already uploaded chunk indexes if an upload session already exists.

2. Client-Side Chunked Uploading

The client iterates over the file in chunk sizes (e.g., 5MB each) and uploads each chunk along with metadata such as uploadId, chunkIndex, and totalChunks.

async function uploadFileInChunks(file) {
    const CHUNK_SIZE = 5 * 1024 * 1024; // 5 MB
    const totalChunks = Math.ceil(file.size / CHUNK_SIZE);
    const fileId = `${file.name}-${file.size}-${file.lastModified}`;

    // Check uploaded chunks from server
    const response = await fetch(`/api/upload/status?fileId=${fileId}`);
    const { uploadedChunks } = await response.json();

    for (let chunkIndex = 0; chunkIndex < totalChunks; chunkIndex++) {
        if (uploadedChunks.includes(chunkIndex)) {
            console.log(`Chunk ${chunkIndex} already uploaded. Skipping...`);
            continue;
        }

        const start = chunkIndex * CHUNK_SIZE;
        const end = Math.min(file.size, start + CHUNK_SIZE);
        const chunk = file.slice(start, end);

        const formData = new FormData();
        formData.append('chunk', chunk);
        formData.append('chunkIndex', chunkIndex);
        formData.append('totalChunks', totalChunks);
        formData.append('fileId', fileId);

        await fetch('/api/upload/chunk', {
            method: 'POST',
            body: formData
        });
    }

    // Notify server to merge chunks
    await fetch('/api/upload/merge', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ fileId, fileName: file.name, totalChunks })
    });
}

3. Server-Side Chunk Handling & Merging

The server saves individual chunk files into a temporary directory named after the fileId. Once all chunks are present, it concatenates them in order into the final file location and removes the temporary files.

const fs = require('fs');
const path = require('path');

async function mergeChunks(fileId, fileName, totalChunks) {
    const tempDir = path.join(__dirname, 'uploads', fileId);
    const targetPath = path.join(__dirname, 'completed', fileName);
    const writeStream = fs.createWriteStream(targetPath);

    for (let i = 0; i < totalChunks; i++) {
        const chunkPath = path.join(tempDir, `chunk-${i}`);
        const chunkBuffer = fs.readFileSync(chunkPath);
        writeStream.write(chunkBuffer);
        fs.unlinkSync(chunkPath);
    }

    writeStream.end();
    fs.rmdirSync(tempDir);
}

Key Best Practices

Concurrency Control: Upload 2 to 4 chunks in parallel to optimize network throughput while avoiding browser connection limits.
Automated Cleanup: Implement background jobs (e.g., cron jobs) to clean up abandoned temporary chunks older than a specified threshold (e.g., 24 hours).
Checksum Verification: Send an MD5 or SHA-256 hash along with each chunk and the final merge request to verify data integrity.
Leverage Cloud Storage: For large scale architectures, leverage native cloud mechanisms like AWS S3 Multipart Upload or Azure Block Blobs directly from the client using pre-signed URLs.