Get started
Yandex Managed Service for Apache Kafka®

Getting started with Managed Service for Apache Kafka®

To get started with the service:

  1. Create a cluster.
  2. Create a topic.
  3. Create an account.
  4. Connect to the cluster.

Before you start

  1. Go to the management console. Then log in to Yandex.Cloud or sign up if you don't have an account yet.

  2. If you don't have a folder yet, create one:

    1. Click Create folder in the Home page of the management console.

    2. Enter the folder name.

      • The name must be unique within the folder.
      • The name may contain lowercase Latin letters, numbers, and hyphens.
      • The first character must be a letter. The last character can't be a hyphen.
      • The maximum length of the name is 63 characters.

    3. Select Create a default network. A network is created with subnets in each availability zone.

    4. Click Create.

You can connect to an Apache Kafka® cluster from both inside and outside Yandex.Cloud:

  • To connect from inside Yandex.Cloud, you need to use a Linux or Windows-based virtual machine, which must be in the same network as the cluster.
  • To connect to a cluster from the internet, enable public access to the cluster when creating it.

Create a cluster

To create a cluster:

  1. In the management console, select the folder where you want to create a cluster.
  2. Select Managed Service for Apache Kafka®.
  3. Click Create cluster.
  4. Set the cluster parameters and click Create cluster. This process is described in detail in Creating clusters.
  5. Wait until the cluster is ready: its status on the Managed Service for Apache Kafka® dashboard changes to Running and its state to Alive. This may take some time.

Then create a topic in the cluster.

Create a topic

Topic is a way to group message streams into categories. Producers write messages to a topic and consumers read messages from it.

To create a topic:

  1. In the management console, select the folder where the cluster is located.
  2. Select Managed Service for Apache Kafka®.
  3. Click on the name of the cluster you created and select the Topics tab.
  4. Click Create topic.
  5. Specify the topic settings and click Create. This process is described in detail in Working with topics and partitions.

Then create an account for producers and consumers.

Create an account

Accounts let you manage producer and consumer permissions to cluster topics.

To create an account:

  1. In the management console, select the folder where the cluster is located.
  2. Select Managed Service for Apache Kafka®.
  3. Click on the name of the cluster you created and select the Users tab.
  4. Click Add.
  5. Enter a name for the account (username) and password (from 8 to 128 characters).
  6. Click Add topic and select the topic you created from the drop-down list.
  7. Add permissions to this topic for the producer and consumer. This process is described in detail in Managing Apache Kafka® accounts.
  8. Click Add.

Then connect to the cluster using this account.

Connect to the cluster

You can connect producers and consumers to clusters using one account. Both the producer and consumer will only be able to work with the topics that they are allowed to access within a specific account.

To connect to a cluster:

  1. Create a Linux VM in the same cloud network as the cluster.

  2. Connect to the VM via SSH.

  3. On the VM, install the kafkacat utility, an open source application that can work as a universal data producer or consumer:

    $ sudo apt-get install kafkacat

  4. Install an SSL certificate on the VM:

    $ sudo mkdir -p /usr/local/share/ca-certificates/Yandex
$ sudo wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" -O /usr/local/share/ca-certificates/Yandex/YandexCA.crt
$ sudo chmod 655 /usr/local/share/ca-certificates/Yandex/YandexCA.crt

  5. To send a message to a topic, run the command:

    $ echo "test message" | kafkacat -P  \
      -b <broker FQDN>:9091 \
      -t <topic name> \
      -X security.protocol=SASL_SSL \
      -X sasl.mechanisms=SCRAM-SHA-512 \
      -X sasl.username=<producer username> \
      -X sasl.password=<producer password> \
      -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexCA.crt  -Z -K:

    In the command, specify the broker FQDN, the topic name, and the username and password of the Apache Kafka® account that you created in the previous step.

    To view an example of the command with the host FQDN filled in, open the cluster page in the management console and click Connect.

  6. To get messages from a topic, run the command:

    $ kafkacat -C \
      -b <broker FQDN>:9091 \
      -t <topic name> \
      -X security.protocol=SASL_SSL \
      -X sasl.mechanisms=SCRAM-SHA-512 \
      -X sasl.username=<consumer username> \
      -X sasl.password=<consumer password> \
      -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexCA.crt -Z -K:

    In the command, specify the broker FQDN, the topic name, and the username and password of the Apache Kafka® account that you created in the previous step.

    To view an example of the command with the host FQDN filled in, open the cluster page in the management console and click Connect.

For more information about connecting to Managed Service for Apache Kafka® clusters, see Connecting to topics in an Apache Kafka® cluster.

What's next

In this article: