Creating clusters

Written by

Examples
- Creating a single-host cluster

A cluster in Managed Service for Apache Kafka^® is one or more broker hosts where topics and their partitions are located. Producers and consumers can work with these topics by connecting to cluster hosts.

Warning

If you create a cluster with more than one host, three dedicated ZooKeeper hosts will be added to the cluster. For more information, see Relationship between service resources.

Management console

CLI

API

Terraform

To create a cluster, follow these steps:

In the management console, select the folder where you want to create a cluster.
Select Managed Service for Apache Kafka^®.
Click Create cluster.
Under Basic parameters:
1. Enter a name for the cluster and, if necessary, a description. The cluster name must be unique within the folder.
2. Select the environment where you want to create the cluster (you can't change the environment once the cluster is created):
  - PRODUCTION: For stable versions of your apps.
  - PRESTABLE: For testing, including the Managed Service for Apache Kafka^® service itself. The Prestable environment is first updated with new features, improvements, and bug fixes. However, not every update ensures backward compatibility.
3. Select the Apache Kafka^® version.
Under Host class, select the platform, host type, and host class.

The host class defines the technical specifications of the VMs where the Apache Kafka^® brokers will be deployed. All available options are listed in Host classes.

When you change the host class for the cluster, the characteristics of all existing instances change.
Under Storage:
1. Choose the type of storage, either a more flexible network type (network-hdd or network-ssd) or faster local SSD storage (local-ssd).
2. Select the size of storage to be used for data. The size of the local storage can only be changed in 100 GB increments.
Under Network settings:
1. Select one or more availability zones where the Apache Kafka^® brokers will reside.
2. Select the network.
3. Select subnets in each availability zone for this network. To create a new subnet, click Create new subnet next to the desired availability zone.
  
  Note
  
  For a cluster with multiple broker hosts, you need to specify subnets in each availability zone even if you plan to host brokers only in some of them. These subnets are required to host three ZooKeeper hosts — one in each availability zone. For more information, see Relationship between service resources.
4. To access broker hosts from the internet, select Public access. In this case, you can only connect to them over an SSL connection. For more information, see Connecting to topics in an Apache Kafka® cluster.
  
  Warning
  
  You can't request public access after creating a cluster.
5. Select security groups to control the cluster's network traffic.
Under Hosts, specify the number of broker hosts Apache Kafka^® to be located in each of the selected availability zones.

When choosing the number of hosts, keep in mind that:
- The Apache Kafka^® cluster hosts will be evenly deployed in the selected availability zones. Decide on the number of zones and hosts per zone based on the required fault tolerance model and cluster load.
- Replication is possible if there are at least two hosts in the cluster.
- Adding more than one host to the cluster automatically adds three ZooKeeper hosts.
If you specify two or more broker hosts, under Host class ZooKeeper, specify the characteristics of the hostsZooKeeper to be located in each of the selected availability zones.
If necessary, configure the Apache Kafka^® settings.
Click Create cluster.
Wait until the cluster is ready: its status on the Managed Service for Apache Kafka^® dashboard changes to Running and its state to Alive. This may take some time.

If you don't have the Yandex.Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a cluster:

View a description of the CLI create cluster command:
```
yc managed-kafka cluster create --help
```

Specify the cluster parameters in the create command (only some of the supported parameters are given in the example):

yc managed-kafka cluster create \
   --name <cluster name> \
   --environment <prestable or production> \
   --version <version: 2.1 or 2.6> \
   --network-name <network name> \
   --brokers-count <number of brokers in the zone> \
   --resource-preset <host class> \
   --disk-type <disk type> \
   --disk-size <storage size in GB> \
   --assign-public-ip <public access> \
   --security-group-ids <list of security group IDs>

If necessary, you can also configure the Apache Kafka^® settings here.

To create a cluster, use the create API method and pass the following in the request:

In the folderId parameter, the ID of the folder where the cluster should be placed.
The cluster name, in the name parameter.
Security group IDs in the parameter securityGroupIds.

With Terraform, you can quickly create a cloud infrastructure in Yandex.Cloud. The infrastructure components are identified through configuration files that specify the required cloud resources and their parameters.

If you don't have Terraform, install it and configure the provider.

To create a cluster:

In the configuration file, describe the parameters of resources that you want to create:

Database cluster: Description of the cluster and its hosts.
Network: Description of the cloud network where the cluster will be located. If you already have a suitable network, you don't need to describe it again.
Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't need to describe them again.

Example configuration file structure:

terraform {
  required_providers {
    yandex = {
     source = "yandex-cloud/yandex"
    }
  }
}

provider "yandex" {
  token = "<OAuth or static key of service account>"
  cloud_id  = "<cloud ID>"
  folder_id = "<folder ID>"
  zone      = "<availability zone>"
}

resource "yandex_mdb_kafka_cluster" "<cluster name>" {
  environment = "<PRESTABLE or PRODUCTION>"
  name        = "<cluster name>"
  network_id  = "<network ID>"
  security_group_ids = ["<list of security groups>"]

  config {
    assign_public_ip = "<public access to the cluster: true or false>"
    brokers_count    = <number of brokers>
    version          = "<Apache Kafka version: 2.1 or 2.6>"
    kafka {
      resources {
        disk_size          = <storage size in GB>
        disk_type_id       = "<storage type: network-ssd, network-hdd, or local-ssd>"
        resource_preset_id = "<host class>"
      }
    }

    zones = [
      "<availability zones>"
    ]
  }
}

resource "yandex_vpc_network" "<network name>" {
  name = "<network name>"
}

resource "yandex_vpc_subnet" "<subnet name>" {
  name           = "<subnet name>"
  zone           = "<availability zone>"
  network_id     = "<network ID>"
  v4_cidr_blocks = ["<range>"]
}

For more information about resources that can be created using Terraform, see the provider documentation.

Make sure that the configuration files are correct.
1. In the command line, go to the folder where you created the configuration file.
2. Run the check using the command:
```
terraform plan
```
If the configuration is described correctly, the terminal displays a list of created resources and their parameters. If there are errors in the configuration, Terraform points them out. This is a test step. No resources are created.
Create a cluster.
1. If the configuration doesn't contain any errors, run the command:
```
terraform apply
```
2. Confirm that you want to create the resources.
After this, all the necessary resources will be created in the specified folder and the IP addresses of the VMs will be displayed in the terminal. You can check resource availability and their settings in management console.

If you don't have Terraform, install it and configure the provider.

To create a cluster:

In the configuration file, describe the parameters of resources that you want to create:

Database cluster: Description of the cluster and its hosts.
Network: Description of the cloud network where the cluster will be located. If you already have a suitable network, you don't need to describe it again.
Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't need to describe them again.

Example configuration file structure:

terraform {
  required_providers {
    yandex = {
     source = "yandex-cloud/yandex"
    }
  }
}

provider "yandex" {
  token = "<OAuth or static key of service account>"
  cloud_id  = "<cloud ID>"
  folder_id = "<folder ID>"
  zone      = "<availability zone>"
}

resource "yandex_mdb_kafka_cluster" "<cluster name>" {
environment = "<PRESTABLE or PRODUCTION>"
name        = "<cluster name>"
network_id  = "<network ID>"

config {
  assign_public_ip = "<public access to the cluster: true or false>"
  brokers_count    = <number of brokers>
  version          = "<Apache Kafka version: 2.1 or 2.6>"
  kafka {
    resources {
      disk_size          = <storage size in GB>
      disk_type_id       = "<storage type: network-ssd, network-hdd, or local-ssd>"
      resource_preset_id = "<host class>"
    }
  }

  zones = [
    "<availability zones>"
    ]
  }
}

resource "yandex_vpc_network" "<network name>" {
  name = "<network name>"
}

resource "yandex_vpc_subnet" "<subnet name>" {
  name           = "<subnet name>"
  zone           = "<availability zone>"
  network_id     = "<network ID>"
  v4_cidr_blocks = ["<range>"]
}

If you don't have Terraform, install it and configure the provider.

To create a cluster:

In the configuration file, describe the parameters of resources that you want to create:

Database cluster: Description of the cluster and its hosts.
Network: Description of the cloud network where the cluster will be located. If you already have a suitable network, you don't need to describe it again.
Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't need to describe them again.

If necessary, you can also configure the Apache Kafka^® settings here.

Example configuration file structure:

terraform {
  required_providers {
    yandex = {
     source = "yandex-cloud/yandex"
    }
  }
}

provider "yandex" {
  token = "<OAuth or static key of service account>"
  cloud_id  = "<cloud ID>"
  folder_id = "<folder ID>"
  zone      = "<availability zone>"
}

resource "yandex_mdb_kafka_cluster" "<cluster name>" {
  environment        = "<PRESTABLE or PRODUCTION>"
  name               = "<cluster name>"
  network_id         = "<network ID>"
  security_group_ids = ["<list of security groups>"]

  config {
    assign_public_ip = "<public access to the cluster: true or false>"
    brokers_count    = <number of brokers>
    version          = "<Apache Kafka version: 2.1 or 2.6>"
    kafka {
      resources {
        disk_size          = <storage size in GB>
        disk_type_id       = "<storage type: network-ssd, network-hdd, or local-ssd>"
        resource_preset_id = "<host class>"
      }
    }

    zones = [
      "<availability zones>"
    ]
  }
}

resource "yandex_vpc_network" "<network name>" {
  name = "<network name>"
}

resource "yandex_vpc_subnet" "<subnet name>" {
  name           = "<subnet name>"
  zone           = "<availability zone>"
  network_id     = "<network ID>"
  v4_cidr_blocks = ["<range>"]
}

For more information about resources that can be created using Terraform, see the provider documentation.

Make sure that the configuration files are correct.
1. In the command line, go to the folder where you created the configuration file.
2. Run the check using the command:
```
terraform plan
```
If the configuration is described correctly, the terminal displays a list of created resources and their parameters. If there are errors in the configuration, Terraform points them out. This is a test step. No resources are created.
Create a cluster.
1. If the configuration doesn't contain any errors, run the command:
```
terraform apply
```
2. Confirm that you want to create the resources.
After this, all the necessary resources will be created in the specified folder and the IP addresses of the VMs will be displayed in the terminal. You can check resource availability and their settings in management console.

Warning

If you specified security group IDs when creating a cluster, you may also need to re-configure security groups to connect to the cluster.

Warning

If you specified security group IDs when creating a cluster, you may also need to re-configure security groups to connect to the cluster.

Examples

Creating a single-host cluster

CLI

Terraform

Let's say we need to create a Managed Service for Apache Kafka^® cluster with the following characteristics:

With the name mykf.
In the production environment.
With Apache Kafka^® version 2.6.
In the default network.
In the security group enp6saqnq4ie244g67sb.
With one s2.micro host in the ru-central1-c availability zone.
With one broker.
With 10 GB fast network storage (network-ssd).
With public access.

Run the command:

yc managed-kafka cluster create \
--name mykf \
--environment production \
--version 2.6 \
--network-name default \
--zone-ids ru-central1-c \
--brokers-count 1 \
--resource-preset s2.micro \
--disk-size 10 \
--disk-type network-ssd \
--assign-public-ip \
--security-group-ids enp6saqnq4ie244g67sb

Let's say we need to create a Managed Service for Apache Kafka^® cluster with the following characteristics:

In the cloud with ID b1gq90dgh25bebiu75o.
In the folder with the ID b1gia87mbaomkfvsleds.
With the name mykf.
In the PRODUCTION environment.
With Apache Kafka^® version 2.6.
In the new mynet network with the subnet mysubnet.
In the new security group mykf-sg allowing connection to the cluster from the Internet via port 9091.
With one s2.micro host in the ru-central1-c availability zone.
With one broker.
With 10 GB fast network storage (network-ssd).
With public access.

The configuration file for the cluster looks like this:

terraform {
  required_providers {
    yandex = {
      source = "yandex-cloud/yandex"
    }
  }
}

provider "yandex" {
  token = "<OAuth or static key of service account>"
  cloud_id  = "b1gq90dgh25bebiu75o"
  folder_id = "b1gia87mbaomkfvsleds"
  zone      = "ru-central1-c"
}

resource "yandex_mdb_kafka_cluster" "mykf" {
  environment = "PRODUCTION"
  name        = "mykf"
  network_id  = yandex_vpc_network.mynet.id
  security_group_ids = [ yandex_vpc_security_group.mykf-sg.id ]

  config {
    assign_public_ip = true
    brokers_count    = 1
    version          = "2.6"
    kafka {
      resources {
        disk_size          = 10
        disk_type_id       = "network-ssd"
        resource_preset_id = "s2.micro"
      }
    }

    zones = [
      "ru-central1-c"
    ]
  }
}

resource "yandex_vpc_network" "mynet" {
  name = "mynet"
}

resource "yandex_vpc_subnet" "mysubnet" {
  name           = "mysubnet"
  zone           = "ru-central1-c"
  network_id     = yandex_vpc_network.mynet.id
  v4_cidr_blocks = ["10.5.0.0/24"]
}

resource "yandex_vpc_security_group" "mykf-sg" {
  name       = "mykf-sg"
  network_id = yandex_vpc_network.mynet.id

  ingress {
    description    = "Kafka"
    port           = 9091
    protocol       = "TCP"
    v4_cidr_blocks = [ "0.0.0.0/0" ]
  }
}