Kafka Internals
How Kafka stores messages in replicated partition logs, and how producers, consumer groups, offsets and transactions behave under failure.
What Kafka is
Apache Kafka is a distributed, replicated, append-only commit log (a file you only add to at the end, never edit). Producers append records; consumers read them at their own pace and can read them again.
This is different from a classic queue (see Message Queues and Pub/Sub), where a message vanishes once consumed. In Kafka the broker keeps the data for a configured time, and each consumer only remembers a position.
- Broker: one Kafka server process. A cluster is several brokers.
- Topic: a named stream, such as
orders. - Partition: one ordered log file set that holds a slice of a topic. A topic has many.
- Record: key, value, timestamp, headers. The key decides the partition.
- Offset: a record's sequence number inside one partition (0, 1, 2...).
Ordering is guaranteed inside a partition only, never across a whole topic.
1 of 11