Get started
Yandex Managed Service for Apache Kafka®

Working with topics and partitions

In Managed Service for Apache Kafka® clusters, you can:

Create a topic

To create a topic, follow these steps:

  1. Go to the folder page and select Managed Service for Apache Kafka®.

  2. Click the name of the cluster and go to the Topics tab.

  3. Click Create topic.

  4. Under Basic parameters, set the basic parameters of the topic:

    • The topic name (must be unique in the Apache Kafka® cluster).
    • The number of topic partitions.
    • The replication factor. This parameter value should not exceed the number of brokers in the cluster. Minimum value: 1. Default:
      • For a cluster with one or two brokers: 1.
      • For a cluster with three or more brokers: 3.

  5. Under Topic settings, specify the topic settings.

  6. Click Create.

Use the create API method and pass the following in the request:

  • In the clusterId parameter, the ID of the cluster where you want to create a topic. To find out the cluster ID, get a list of clusters in the folder.
  • Topic settings in the topicSpec parameter.

Update topic settings

Warning

The number of Apache Kafka® topic partitions can only be increased.

To change topic settings, follow these steps:

  1. Go to the folder page and select Managed Service for Apache Kafka®.
  2. Click on the name of the cluster and select the Topics tab.
  3. Click for the necessary topic and select Edit.
  4. Change the basic parameters of the topic:
    • The number of topic partitions.
    • The replication factor. This parameter value should not exceed the number of brokers in the cluster. Minimum value: 1. Default:
      • For a cluster with one or two brokers: 1.
      • For a cluster with three or more brokers: 3.
  5. Change the topic settings.
  6. Click Save.

Use the update API method and pass the following in the request:

  • In the clusterId parameter, the ID of the cluster where the topic is located. To find out the cluster ID, get a list of clusters in the folder.
  • The topic name in the topicName parameter. To find out the name, get a list of topics in the cluster.
  • In the updateMask parameter, a list of settings to update (in a single line, comma-separated). If this parameter is omitted, the API method resets any topic settings that aren't explicitly specified in the request to their default values.
  • New values for the topic settings in the topicSpec parameter.

Delete a topic

Note

Permissions granted to the account for a topic are retained even after the topic is deleted. You can either revoke such permissions or leave them as is. In the latter case, you can create a topic with the same name and use it without reassigning account rights.

To delete a topic, follow these steps:

  1. Go to the folder page and select Managed Service for Apache Kafka®.
  2. Click the name of the cluster and go to the Topics tab.
  3. Click for the necessary topic and select Delete topic.
  4. Confirm topic deletion and click Delete.

Use the delete API method and pass the following in the request:

Get a list of topics in a cluster

To get a list of topics, follow these steps:

  1. Go to the folder page and select Managed Service for Apache Kafka®.
  2. Click the name of the cluster and go to the Topics tab.

Use the list API method: pass the ID of the required cluster in the clusterId request parameter.

To find out the cluster ID, get a list of clusters in the folder.

Get detailed information about a topic

To get detailed information about a topic, follow these steps:

  1. Go to the folder page and select Managed Service for Apache Kafka®.
  2. Click the name of the cluster and go to the Topics tab.
  3. Click on the name of the topic.

To get detailed information about a topic, use the get API method and pass the following in the request:

Additional topic settings

These settings specify various topic parameters:

  • Cleanup policy:

    • Delete: Delete log segments when either their retention time or log size limit is reached.
    • Compact: Compact messages in the log.
    • CompactAndDelete: Both compact messages and delete log segments.

    For more information, see the documentation for Apache Kafka®.

  • Compression type: The codec to use for message compression:

    • Uncompressed: No compression.
    • Zstd: The Zstandard codec.
    • Lz4: The LZ4 codec.
    • Snappy: The Snappy codec.
    • Gzip: The GZip codec.
    • Producer: The codec is set by the producer.

    For more information, see the documentation for Apache Kafka®.

  • Delete retention, ms: The amount of time in milliseconds to retain delete tombstone markers for log compacted topics. You can see this parameter only when you select the Compact and CompactAndDelete cleanup policies.

    For more information, see the documentation for Apache Kafka®.

  • Delete delay, ms: The time to wait before deleting a file from the filesystem.

    For more information, see the documentation for Apache Kafka®.

  • Maximum number of messages in memory: The number of topic messages that can be kept in memory before these messages are flushed to disk. For example, if the parameter is 1, the disk is flushed after each message is received. If it's set to 5, messages are flushed to disk in groups of five.

    For more information, see the documentation for Apache Kafka®.

  • Message retention time, ms: The time that a message in any topic is kept in memory before it's flushed to disk.

    For more information, see the documentation for Apache Kafka®.

  • Minimum compaction lag, ms: The minimum time a message will remain uncompacted in the log.

    For more information, see the documentation for Apache Kafka®.

  • Maximum partition size, bytes: The maximum size that a partition can grow to. After reaching this size, Apache Kafka® starts deleting old log segments. This setting applies if the Delete log cleanup policy is in effect.

    It's helpful if you need to control the size of a log due to limited disk space.

    For more information, see the documentation for Apache Kafka®.

  • Log segment lifetime, ms: The time after which Apache Kafka® starts deleting old log segments. This setting applies if the Delete log cleanup policy is in effect.

    For more information, see the documentation for Apache Kafka®.

  • Maximum batch size: The maximum batch size of messages that Apache Kafka® allows you to write to a topic to the producer or read from a topic to the consumer (in bytes, after compression, if enabled).

    Minimum value: 0 (default: 1,048,588, 1 MB).

    For more information, see the documentation for Apache Kafka®.

  • Minimum number of in-sync replicas: The minimum number of replicas to wait for record confirmation from in order to consider a message successfully written to a topic. This setting is useful when the broker is waiting for record confirmation from all replicas in the cluster.

    Minimum value: 1. Default:

    • For a cluster with one or two brokers: 1.
    • For a cluster with three or more brokers: 2.

    For more information, see the documentation for Apache Kafka®.

In this article: