Disk types in Managed Service for Apache Kafka®
Managed Service for Apache Kafka® allows you to use network and local storage drives for database clusters. Network storage drives are based on network blocks, which are virtual disks in the Yandex Cloud infrastructure. Local disks are physically located on broker servers.
When creating a cluster, you can select the following disk types for data storage:
-
Network HDD storage (
network-hdd
): Most cost-effective option for clusters that do not require high read/write performance. -
Network SSD storage (
network-ssd
): Balanced solution. Such disks are slower than local SSD storage, but, unlike local disks, they ensure data integrity in case Yandex Cloud hardware goes down. -
Non-replicated SSD storage (
network-ssd-nonreplicated
): Network SSD storage with enhanced performance but without redundancy.The storage size can only be increased in 93 GB increments.
-
Local SSDs (
local-ssd
): Disks with the fastest performance.The size of such a storage can be increased:
-
For Intel Cascade Lake: In 100 GB increments only.
-
For Intel Ice Lake: In 368 GB increments only.
Note
For clusters with hosts residing in the
ru-central1-d
availability zone, local SSD storage is not available if using the Intel Cascade Lake platform. -
Selecting disk type during cluster creation
The number of broker hosts that can be created along with an Apache Kafka® cluster depends on the selected disk type:
-
With local SSD (
local-ssd
) or non-replicated SSD (network-ssd-nonreplicated
) storage, you can create a cluster with three or more broker hosts.This cluster will be fail-safe only if it meets all fault tolerance conditions.
-
With network HDD (
network-hdd
) or network SSD (network-ssd
) storage, you can add any number of broker hosts within the current quota.
For more information about limits on the number of broker hosts per cluster, see Quotas and limits.
Minimum storage size
In order to work, each topic requires space in broker host storage. The amount of such space depends on the replication factor and the number of partitions. If there is not enough available storage space, you will not be able to create a new topic.
You can calculate the minimum storage size for all topics using the formula below:
2 × maximum log segment size × number of partitions in cluster × replication factor
If topic partitions are evenly distributed, divide the value calculated with this formula by the number of broker hosts.
Maximum log segment size
At least two log segments are required for each replica of a topic partition. You can set the maximum size of such a segment:
- At the topic level using the Segment bytes setting.
- Globally at the cluster level using the Log segment bytes setting.
The minimum storage size for all topics is: 2 × maximum log segment size × number of partitions per cluster × replication factor
. If the cluster partitions are evenly distributed, you can divide the resulting value by the number of brokers to determine the required storage size per broker.
By default, the segment size is 1 GB.