Apache Kafka can easily process millions of messages per second per server. This exceptional performance is achieved through key architectural decisions at the storage and operating system levels.
1. Sequential I/O Disk Access
Rather than performing random memory or disk access, Kafka appends data sequentially to the end of partition log files. Sequential disk reads and writes are significantly faster than random access, rivaling random memory performance.
2. Page Cache Utilization
Kafka relies heavily on the operating system's Page Cache instead of maintaining a large in-memory heap inside the JVM process. This reduces garbage collection overhead and maximizes memory usage.
3. Zero-Copy Transfer
Kafka utilizes the OS sendfile system call to transfer data directly from the OS Page Cache to the network socket without copying data into user space application memory.
// Conceptual representation of Zero-Copy OS transfer
// Traditional: Disk -> OS Page Cache -> Application Buffer -> Socket Buffer -> Network
// Zero-Copy: Disk -> OS Page Cache -----------------------> Socket Buffer -> Network4. Batching and Compression
Producers batch multiple records into a single request, compressing them before sending across the network, which minimizes network bandwidth usage and system call overhead.
