Getting started with Managed Service for Apache Kafka®
To get started with the service:
Getting started
-
Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. -
If you do not have a folder yet, create one:
-
In the management console
, select the appropriate cloud in the list on the left. -
At the top right, click
-
Enter the folder name. The naming requirements are as follows:
- The name must be from 3 to 63 characters long.
- It may contain lowercase Latin letters, numbers, and hyphens.
- The first character must be a letter and the last character cannot be a hyphen.
-
(Optional) Enter a description of the folder.
-
Select Create a default network. This will create a network with subnets in each availability zone. Within this network, a default security group will be created, inside which all network traffic is allowed.
-
Click Create.
-
-
You can connect to an Apache Kafka® cluster from both inside and outside Yandex Cloud:
-
To connect from inside Yandex Cloud, create a Linux- virtual machine, which must be in the same network as the cluster.
-
To connect to a cluster from the internet, enable public access to the cluster when creating it.
Note
The next step assumes that you connect to the cluster from a Linux-based VM.
-
-
Connect to the VM over SSH.
-
Install the
kafkacat
utility, an open source application that can function as a universal data producer or consumer:sudo apt-get install kafkacat
Create a cluster
To create a cluster:
- In the management console, select the folder where you want to create a cluster.
- Select Managed Service for Kafka.
- Click Create cluster.
- Set the cluster parameters and click Create. For more information, see Creating clusters.
- Wait until the cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to
Running
and its state toAlive
. This may take some time.
Then create a topic in the cluster.
Create the topic
A topic is a way to group message streams into categories. Producers write messages to a topic and consumers read messages from it.
To create a topic:
- In the management console, select the folder where the cluster is located.
- Select Managed Service for Kafka.
- Click the name of the cluster you created and select the Topics tab.
- Click Create topic.
- Specify the topic settings and click Create. This process is described in detail in Working with topics and partitions.
Then create users for producers and consumers.
Create a user
User settings let you manage producer and consumer permissions to cluster topics.
To create a user:
- In the management console, select the folder where the cluster is located.
- Select Managed Service for Kafka.
- Click the name of the cluster you created and select the Users tab.
- Click Add.
- Enter a username and password (from 8 to 128 characters).
- Click
- Add permissions to this topic for the producer and consumer. For more information, see User management.
- Click Add.
Then connect to the cluster using this username.
Connect to the cluster
You can connect the producer and consumer to the cluster on behalf of one user. Both the producer and consumer will only be able to work with the topics that this user is allowed to access.
To connect to a cluster:
-
If you are using security groups for a cloud network, configure them to enable all relevant traffic between the cluster and the connecting host.
-
Install an SSL certificate on the VM:
Linux (Bash)/macOS (Zsh)Windows (PowerShell)mkdir -p /usr/local/share/ca-certificates/Yandex/ && \ wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \ --output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \ chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
The certificate will be saved to the
/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
file.mkdir $HOME\.kafka; curl.exe -o $HOME\.kafka\YandexInternalRootCA.crt https://storage.yandexcloud.net/cloud-certs/CA.pem
The certificate is saved to the
$HOME\.kafka\YandexInternalRootCA.crt
file. -
To send a message to a topic, run the command:
echo "test message" | kafkacat -P \ -b <broker_FQDN>:9091 \ -t <topic_name> \ -k key \ -X security.protocol=SASL_SSL \ -X sasl.mechanism=SCRAM-SHA-512 \ -X sasl.username="<producer_login>" \ -X sasl.password="<producer_password>" \ -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z
In the command, specify the broker FQDN, the topic name, and the username and password of the Apache Kafka® user that you created in the previous step.
Learn how to get the FQDN of a broker host in this guide.
-
To get messages from a topic, run the following command:
kafkacat -C \ -b <broker_FQDN>:9091 \ -t <topic_name> \ -X security.protocol=SASL_SSL \ -X sasl.mechanism=SCRAM-SHA-512 \ -X sasl.username="<consumer_username>" \ -X sasl.password="<consumer_password>" \ -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z -K:
In the command, specify the broker FQDN, the topic name, and the username and password of the Apache Kafka® user that you created in the previous step.
Learn how to get the FQDN of a broker host in this guide.
For more information about connecting to a Managed Service for Apache Kafka® cluster, see Connecting to topics in a cluster.
What's next
- Read about service concepts.
- Learn more about creating a cluster and connecting to a cluster.