What is Kafka Log Compaction and When Should You Use It?

Kafka topics typically retain messages for a fixed period based on time (e.g., 7 days) or log size. However, for state-restoration or key-value lookup scenarios, Log Compaction offers a specialized retention strategy.

How Log Compaction Works

Log compaction ensures that Kafka retains at least the last known value for each message key within the log of a data topic partition.

• Head (Clean): Contains newly written messages; functions like a normal topic log.
• Tail (Dirty): Contains older messages where duplicate keys are removed by the background log cleaner threads.

Enabling Log Compaction

# Enable log compaction on topic creation
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic user-profiles \
  --partitions 3 \
  --replication-factor 2 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.5

Use Cases

Log compaction is ideal for scenarios such as restoring application state in KTable operations, maintaining system configuration states, or keeping track of active database records in Change Data Capture (CDC) pipelines.