Topics and partitions
A topic is a way to group message streams into categories. Producers publish messages of a certain category in a topic, while consumers subscribe to this topic and read messages from it. For each topic, Apache Kafka® keeps a message log that can be divided into several partitions.
For example, if the data producer is an online store, you can create separate topics for logging user actions, storing data about their shopping cart, keeping records about transactions, and so on.
Managed Service for Apache Kafka® manages storage of topic messages and enables:
- Partition replication: provided that a cluster consists of at least two brokers and the replication factor set for topics is greater than one.
- Message compaction.
- Log purging in accordance with the policy when partition messages become outdated or the specified log size is reached.
A partition is a sequence of topic messages that are stored in the order they are received. The message sequence number is called an offset. Consumers read messages from the beginning to the end of a partition, that is, messages that were received earlier are consumed first. To start reading data from a specific message, consumers must pass the broker its offset.
Using partitions, you can:
- Distribute the message storage and request processing load among multiple brokers Apache Kafka®.
- Provide fault tolerance: a partition can be replicated to a specified number of brokers.
For more information about topics and partitions, see the documentation for Apache Kafka®.
Managing topics and partitions
You select a topic management method once when creating a cluster and cannot change it afterwards.
A Managed Service for Apache Kafka® cluster provides two ways for you to manage topics and partitions:
Using Yandex.Cloud standard interfaces (CLI, API, or management console). Select this method if you wish to create, delete, and configure topics and partitions using Managed Service for Apache Kafka® service features.
Using the Apache Kafka® Admin API. Select this method if you already have a solution for managing topics and partitions or if you do not plan to use Yandex.Cloud interfaces. Restrictions apply when using the Admin API:
You cannot create or delete topics or manage them using standard Yandex.Cloud interfaces. Use the administrator account instead. This account includes the
ACCESS_ROLE_ADMINrole that applies to all the topics in a cluster.
For information on creating this account, see Managing Apache Kafka® accounts.
You can configure permissions for producer and consumer accounts using standard Yandex.Cloud interfaces rather than the Apache Kafka® Admin API.