How Does Apache Kafka Guarantee Message Ordering Across Partitions?

Question

0

A common question when designing event-driven systems with Apache Kafka is how to guarantee that messages are processed in the exact order they were produced.

Partition-Level Ordering Guarantee

Kafka guarantees strict message ordering only within a single partition, not across all partitions of a topic. If order matters for a specific entity (such as user transactions or account updates), all records for that entity must be routed to the same partition.

Using Message Keys

By specifying a key when producing records, Kafka's default partitioner hashes the key to consistently assign it to the same partition index.

ProducerRecord<String, String> record = new ProducerRecord<>(
    "account-transactions",
    "account-12345",
    "{\"amount\": 250.00, \"type\": \"DEPOSIT\"}"
);

producer.send(record, (metadata, exception) -> {
    if (exception == null) {
        System.out.println("Message sent to partition: " + metadata.partition());
    } else {
        exception.printStackTrace();
    }
});

Key Considerations for Ordering

Max In-Flight Requests: Ensure max.in.flight.requests.per.connection=1 or enable idempotence (enable.idempotence=true) to prevent message reordering during retries.
Partition Count Changes: Adding partitions to an existing topic will change key hashing distribution, potentially routing existing keys to new partitions.

asked by Uttam Misra , modified by Uttam Misra in Apache Kafka

1 Answer

Write Your Answer

Answer 1

Understanding Kafka Message Ordering Guarantees

By default, Apache Kafka guarantees message ordering only within a single partition. It does not provide out-of-the-box global message ordering across multiple partitions in a topic. This architectural design allows Kafka to achieve massive horizontal scalability, parallel processing, and high throughput across clusters.

1. Partition-Level Ordering

Each partition in a Kafka topic functions as an append-only commit log. When a producer writes records to a specific partition, Kafka assigns them sequential, monotonically increasing offset numbers. A consumer reading from that partition will always consume messages in the exact order they were written.

2. Achieving Entity-Level Ordering with Message Keys

Although Kafka does not guarantee global ordering across partitions, most real-world applications only require ordering for specific entities (such as events related to a specific user ID or order ID). You can achieve strict ordering for related events by assigning a message key.

ProducerRecord<String, String> record = new ProducerRecord<>(
    "user-events",
    "user_12345",
    "User account balance updated"
);
producer.send(record);

Kafka's default partitioner hashes the record's key to select the destination partition. Because identical keys produce the same hash value, all events associated with user_12345 will land in the exact same partition, guaranteeing chronological sequence delivery.

3. Achieving Global Ordering via Single-Partition Topics

If your system strictly demands global ordering across all events in a topic, you can set the topic's partition count to 1. However, this creates a performance bottleneck because only one consumer thread in a consumer group can read from that topic at any given time, disabling horizontal scalability.

4. Preventing Ordering Issues from Retries and Network Failures

When producers retry failed requests due to transient network issues, messages can potentially arrive out of order at the broker. To preserve ordering during retries without sacrificing throughput, enable idempotent producers:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// Guarantees exact order and prevents duplicate writes on retries
props.put("enable.idempotence", "true");
props.put("max.in.flight.requests.per.connection", "5");

Summary

Single Partition: Guarantees strict ordering for all records in the partition.
Across Partitions: No global order guarantee exists by default.
Keyed Partitioning: Guarantees relative ordering for messages sharing the same key.
Idempotent Producer: Prevents out-of-order writes during network retries.