How Does Kafka Achieve High Throughput and Low Latency?

Question

1

Apache Kafka can easily process millions of messages per second per server. This exceptional performance is achieved through key architectural decisions at the storage and operating system levels.

1. Sequential I/O Disk Access

Rather than performing random memory or disk access, Kafka appends data sequentially to the end of partition log files. Sequential disk reads and writes are significantly faster than random access, rivaling random memory performance.

2. Page Cache Utilization

Kafka relies heavily on the operating system's Page Cache instead of maintaining a large in-memory heap inside the JVM process. This reduces garbage collection overhead and maximizes memory usage.

3. Zero-Copy Transfer

Kafka utilizes the OS sendfile system call to transfer data directly from the OS Page Cache to the network socket without copying data into user space application memory.

// Conceptual representation of Zero-Copy OS transfer
// Traditional: Disk -> OS Page Cache -> Application Buffer -> Socket Buffer -> Network
// Zero-Copy:   Disk -> OS Page Cache -----------------------> Socket Buffer -> Network

4. Batching and Compression

Producers batch multiple records into a single request, compressing them before sending across the network, which minimizes network bandwidth usage and system call overhead.

1 Answer

Write Your Answer

Answer 1

Apache Kafka is widely recognized for its ability to process millions of messages per second with sub-millisecond latency. This remarkable performance is not due to a single breakthrough, but rather a combination of architectural decisions and operating system optimizations. Below are the key mechanisms that enable Kafka to achieve high throughput and low latency.

1. Sequential I/O on Disk

A common misconception is that disk storage is inherently too slow for high-performance messaging systems. While random disk access is indeed slow due to seek overhead, sequential disk I/O is extremely fast—often rivaling or exceeding random memory access speeds. Kafka appends messages to an immutable commit log sequentially, eliminating expensive disk seeks and taking full advantage of modern hard drives and NVMe storage.

2. Zero-Copy Technology

In traditional data transmission, transferring data from a file to a network socket involves multiple buffer copies between kernel space and user space. Kafka leverages the OS kernel's zero-copy optimization using the sendfile system call. This allows data to be transferred directly from the OS page cache to the network NIC (Network Interface Card) buffer, completely bypassing user-space copy operations and drastically reducing CPU context switches.

3. Page Cache Instead of JVM Heap

Kafka relies heavily on the operating system's Page Cache rather than maintaining an in-memory cache inside the Java Virtual Machine (JVM) heap. By offloading caching to the OS kernel:

Kafka avoids JVM Garbage Collection (GC) pauses that degrade latency under high throughput.
Cached pages remain available even if the Kafka process restarts.
Memory utilization is maximized dynamically by the operating system.

4. Producer Batching and Compression

To reduce network call overhead, Kafka producers group individual messages into batches before sending them over the wire. Instead of issuing a network request for every single message, a batch of messages is transmitted together. Furthermore, Kafka supports efficient end-to-end compression algorithms (such as Snappy, LZ4, or Zstandard) on these batches, significantly reducing network bandwidth and disk storage footprint.

// Example Java configuration to tune Kafka Producer for high throughput
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// Tune batching and compression parameters
props.put("linger.ms", 20);           // Wait up to 20ms to accumulate messages in a batch
props.put("batch.size", 65536);        // Increase batch size to 64 KB
props.put("compression.type", "snappy"); // Enable snappy compression for low CPU overhead

5. Partitioning and Parallelism

Kafka topics are divided into multiple partitions distributed across a cluster of brokers. Each partition acts as an independent commit log that can be written to and read from concurrently. This architecture allows horizontal scaling: adding more partitions and brokers directly increases write throughput and enables consumer groups to process data in parallel.

Summary

Kafka achieves its impressive speed by combining low-level OS capabilities like sequential I/O and zero-copy data transfer with smart application design like batching, compression, and partitioning.