Topics and partitions

Topics

A topic is a way to group message streams into categories. Producers publish messages of a certain category in a topic, while consumers subscribe to this topic and read messages from it. For each topic, Apache Kafka^® keeps a message log that can be divided into several partitions.

For example, if the data producer is an online store, you can create separate topics for logging user actions, storing data about their shopping cart, keeping records about transactions, and so on.

Managed Service for Apache Kafka^® manages the storage of topic messages and enables:

• Partition replication: provided that a cluster consists of at least two brokers and the replication factor set for topics is greater than one.
• Message compaction.
• Log purging in accordance with the policy when partition messages become outdated or the specified log size is reached.

Partitions

A partition is a sequence of topic messages that are stored in the order they are received. The message sequence number is called an offset. Consumers read messages from the beginning to the end of a partition, that is, messages that were received earlier are consumed first. To start reading data from a specific message, consumers must pass the broker its offset.

Using partitions, you can:

• Distribute the message storage and request processing load among multiple brokers Apache Kafka^®.
• Provide fault tolerance: a partition can be replicated to a specified number of brokers.

For more information about topics and partitions, see the documentation for Apache Kafka^®.