Resource relationships in Managed Service for Apache Kafka®
Yandex Managed Service for Apache Kafka® helps you deploy and maintain clusters of 2.8, 3.0, 3.1, and 3.2 Apache Kafka® servers in the Yandex Cloud infrastructure.
Apache Kafka® is a distributed cross-application communication system that uses the publication-subscription principle.
Each Managed Service for Apache Kafka® cluster consists of one or more broker hosts, where topics and topic partitions are located. Cluster hosts may reside in different availability zones. You can learn more about Yandex Cloud availability zones here.
If all broker hosts are created in the same availability zone, you cannot change their number.
Cluster hosts accept connections from clients, such as data producer and consumer applications. Producers send messages to certain cluster topics and consumers read messages from them. This makes sure you will not need to re-configure consumers if the producer changes.
Depending on the number of broker hosts in the cluster, ZooKeeper is automatically configured and hosted:
- If the Apache Kafka® cluster consists of one broker host, ZooKeeper is hosted on the same host.
- If the Apache Kafka® cluster consists of two or more hosts, ZooKeeper is hosted separately from brokers on three additional hosts. These hosts are added to the cluster automatically.
You cannot delete ZooKeeper hosts. The number of ZooKeeper hosts is permanent. Managed Service for Apache Kafka® uses ZooKeeper to store cluster statuses and configurations.
You can change the ZooKeeper host settings using the CLI.
Note
ZooKeeper hosts get the following specs by default:
- Host class:
b2.medium
- Disk size: 10 GB
- Disk type: Fast network disks
VMs for cluster hosts can be hosted on:
-
Regular Yandex Cloud hosts:
These are physical servers for hosting cluster VMs. They are randomly selected from a pool of available hosts that meet the selected cluster configuration.
-
Dedicated Yandex Cloud hosts:
These are physical servers that only host your VMs. Such VMs ensure the operation of both the cluster and your other services that support dedicated hosts. The hosts are selected from dedicated host groups specified when creating a cluster.
A cluster with multiple broker hosts needs at least three groups of dedicated hosts for its operation. This is required for deploying ZooKeeper hosts.
Such a placement option makes sure the VMs are physically isolated. A Managed Service for Apache Kafka® cluster using dedicated hosts includes all features of a regular cluster.
For more information, see Dedicated hosts.
For more information, see the Apache Kafka® documentation
Cluster fault tolerance
To make sure the cluster data is stored in a reliable way and continuously available:
-
Your cluster must have at least three broker hosts.
-
Each of the three availability zones must host at least one broker host.
-
Cluster topics must have the following parameters:
- Replication factor:
3
- Minimum number of in-sync replicas:
2
- Replication factor:
For more info on the cluster fault tolerance, see the Service Level Agreement (SLA)