Yandex.Cloud
  • Services
  • Why Yandex.Cloud
  • Pricing
  • Documentation
  • Contact us
Get started
Yandex Managed Service for Apache Kafka®
  • Getting started
  • Step-by-step instructions
    • All instructions
    • Information about existing clusters
    • Creating clusters
    • Connecting to clusters
    • Stopping and starting clusters
    • Changing cluster settings
    • Working with topics and partitions
    • Managing Kafka accounts
    • Deleting clusters
  • Concepts
    • Relationship between service resources
    • Topics and partitions
    • Brokers
    • Producers and consumers
    • Host classes
    • Network in Managed Service for Apache Kafka®
    • Quotas and limits
    • Storage types
  • Access management
  • Pricing policy
  • API reference
    • Authentication in the API
    • gRPC
      • Overview
      • ClusterService
      • ResourcePresetService
      • TopicService
      • UserService
      • OperationService
    • REST
      • Overview
      • Cluster
        • Overview
        • create
        • delete
        • get
        • list
        • listHosts
        • listLogs
        • listOperations
        • move
        • start
        • stop
        • streamLogs
        • update
      • ResourcePreset
        • Overview
        • get
        • list
      • Topic
        • Overview
        • create
        • delete
        • get
        • list
        • update
      • User
        • Overview
        • create
        • delete
        • get
        • grantPermission
        • list
        • revokePermission
        • update
      • Operation
        • Overview
        • get
  • Questions and answers
  1. Concepts
  2. Topics and partitions

Topics and partitions

  • Topics
  • Partitions

Topics

A topic is a way to group message streams into categories. Producers publish messages of a certain category in a topic, while consumers subscribe to this topic and read messages from it. For each topic, Apache Kafka® keeps a message log that can be divided into several partitions.

For example, if the data producer is an online store, you can create separate topics for logging user actions, storing data about their shopping cart, keeping records about transactions, and so on.

Managed Service for Apache Kafka® manages the storage of topic messages and enables:

  • Partition replication: provided that a cluster consists of at least two brokers and the replication factor set for topics is greater than one.
  • Message compaction.
  • Log purging in accordance with the policy when partition messages become outdated or the specified log size is reached.

Partitions

A partition is a sequence of topic messages that are stored in the order they are received. The message sequence number is called an offset. Consumers read messages from the beginning to the end of a partition, that is, messages that were received earlier are consumed first. To start reading data from a specific message, consumers must pass the broker its offset.

Using partitions, you can:

  • Distribute the message storage and request processing load among multiple brokers Apache Kafka®.
  • Provide fault tolerance: a partition can be replicated to a specified number of brokers.

For more information about topics and partitions, see the documentation for Apache Kafka®.

In this article:
  • Topics
  • Partitions
Language
Careers
Privacy policy
Terms of use
© 2021 Yandex.Cloud LLC