Get started
Yandex Managed Service for Kubernetes

Kubernetes cluster backups in Yandex Object Storage

Data in Managed Service for Kubernetes clusters is stored securely and replicated within the Yandex.Cloud infrastructure. However, whenever you want, you can back up data from Kubernetes cluster node groups and store them in Object Storage or other storage.

You can back up:

  • Data from Kubernetes cluster node groups using Velero.

  • Persistent volumes using restic. Learn more about configuring restic in the Velero documentation.

    restic copies data from Persistent Volumes as files, so we recommend backing up the database using the application itself. For example, for the ClickHouse DBMS, use the clickhouse-backup utility, for the MySQL DBMS, the MySQLdump utility, for PostgreSQL, pg_dump.

Requirements

The Kubernetes cluster node group must have internet access. You can only grant internet access during node group creation.

Backups

To back up cluster group data:

  1. Choose the latest version of Velero for your platform.

  2. Download Velero, unpack the archive, and install the program. Learn more about installing the program in the Velero documentation.

  3. View a description of any Velero command using the --help flag. For example:

    velero --help

  4. In Object Storage, create a bucket that will store the backup:

    1. In management console, select the folder where you want to create a bucket.
    2. Select Object Storage.
    3. Click Create bucket.
    4. On the New bucket page:

      1. Enter the bucket Name following the naming conventions.

        For example, velero-backup.

      2. If necessary, limit the Maximum size of the bucket.

        If 0, the maximum size isn't limited. It's similar to the enabled No limit option.

      3. Select the type of Access.

      4. Select the default storage class: Standard.

      5. Click Create bucket to complete the operation.

  5. Create a service account that bucket access will be assigned to:

    1. In management console, select the folder where you want to create a service account.

    2. Go to the Service accounts tab.

    3. Click Create service account.

    4. Enter a service account Name. For example, velero-sa.

      • Length — from 3 to 63 characters.
      • The name may contain lowercase Latin letters, numbers, and hyphens.
      • The first character must be a letter. The last character can't be a hyphen.

    5. Click Create.

    If you don't have the Yandex.Cloud command line interface yet, install and initialize it.

    The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

    Create a service account:

    yc iam service-account create --name velero-sa

    Where:

    • name: Service account name.

    Execution result:

    id: abcdo12h3j04odg56def
folder_id: b1g23ga45ghf0cljderg
created_at: "2020-10-19T12:59:56Z"
name: velero-sa

  6. Create a static access key for the service account velero-sa:

    1. In management console, select the folder that the velero-sa service account pertains to.

    2. Go to the Service accounts tab.

    3. Choose the velero-sa service account and click the line with its name.

    4. On the Overview page, click Create new key in the top panel.

    5. Click Create static access key.

    6. In the New key window that opens, click Create.

    7. Save the ID and private key.

      Alert

      After you close the window, the private key value will be unavailable.

    1. Create a static access key for your service account:

      yc iam access-key create --service-account-name velero-sa

      Where:

      • service-account-name: Service account name.

      Execution result:

      access_key:
  id: abcdo12h3j04odg56def
  service_account_id: ajego12h3j03slk16upe
  created_at: "2020-10-19T13:22:29Z"
  key_id: y1qiM23o-Y3WeoP5oSdc
secret: MLSeE12TYJZpjFkfgMeKJ3e7PR7z6dk3UyEeC7PJ

    2. Save the ID key_id and private key secret. You will not be able to get the key value again.

  7. Create a file with a static access key.

    For example, you can create a file with the credentials filename and save it in the home directory.

    To create static access keys that are compatible with the AWS API, the file contents must look as follows:

    [default]
aws_access_key_id=y1qiM23o-Y3WeoP5oSdc
aws_secret_access_key=MLSeE12TYJZpjFkfgMeKJ3e7PR7z6dk3UyEeC7PJ

  8. Grant the velero-sa service account access to the bucket velero-backup. Object Storage supports access control using ACL.

    1. In management console, select the folder where the velero-backup bucket was created.
    2. Select Object Storage.
    3. To edit the ACL, click to the right of the bucket name and select Bucket ACL.
    4. In the ACL editing window that opens:
      1. Select the velero-sa user.
      2. Grant the user READ + WRITE permission.
    5. Click Add.
    6. Click Save.

  9. Install Velero in the Kubernetes cluster where the data needs to be backed up from:

    velero install \
  --backup-location-config s3Url=https://storage.yandexcloud.net,region=ru-central1 \
  --bucket velero-backup \
  --plugins velero/velero-plugin-for-aws:v1.0.0 \
  --provider aws \
  --secret-file ./credentials \
  --use-restic \
  --use-volume-snapshots false

    Where:

    • backup-location-config: Configuration of the backup storage address. URL of Object Storage storage and availability zones.
    • bucket: Name of the backup storage bucket.
    • plugins: Image plugin to enable compatibility with AWS API.
    • provider: Name of the Object Storage object storage provider.
    • secret-file: Name of the file with a static access key and the path to it.
    • use-restic: Parameter for backing up Persistent Volumes with restic.
    • use-volume-snapshots: Parameter to prevent automatically creating storage snapshots.

    Execution result:

    CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: already exists, proceeding
CustomResourceDefinition/backups.velero.io: created
...
DaemonSet/restic: already exists, proceeding
DaemonSet/restic: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

  10. Back up data from the Kubernetes cluster node group:

    velero backup create my-backup --default-volumes-to-restic

    Where:

    • default-volumes-to-restic: Copying Persistent Volumes from the default namespace with restic.

    Execution result:

    Backup request "my-backup" submitted successfully.
Run `velero backup describe my-backup` or `velero backup logs my-backup` for more details.

  11. Wait for the backup to complete. The STATUS will be Completed.

    velero backup get

    Execution result:

    NAME        STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
my-backup   Completed   0        0          2020-10-19 17:13:25 +0300 MSK   29d       default            <none>

Restoring data from backups

To restore data from the Kubernetes cluster node group:

  1. Create a Kubernetes cluster.

    How to create a cluster Kubernetes
    1. In management console, select the folder where you want to create your Kubernetes cluster.
    2. In the list of services, select Managed Service for Kubernetes.
    3. Click Create cluster.
    4. Enter a name and description for the Kubernetes cluster.
    5. Specify a service account for the resources. This is used to create the resources.
    6. Specify a service account for nodes. The nodes use this service account to access the Docker image registry.
    7. Specify a release channel.
    8. Under Master configuration:

      • In the Kubernetes version field, select the Kubernetes version to be installed on the master.

      • In the Public IP field, choose a method for assigning an IP address:

        • Auto: Assign a random IP address from the Yandex.Cloud IP pool.
        • No address: Don't assign a public IP address.

      • In the Master type field, select the type of master:

        • Zonal: A master created in a subnet in one availability zone.
        • Regional: A master created and distributed in three subnets in each availability zone.

      • Select the availability zone where you want to create the master node address.

        This step is only available for the zonal master.

      • In the Network format field, choose how networks are displayed:

        • List: Available networks are displayed as a list. If you don't have a cloud network, click Create network:
          • In the window that opens, enter a name for the network. Specify that subnets should be created. Click Create network. By default, networks are created with subnets in each availability zone.
        • ID: Enter the unique ID of the required networks.

      • In the Cloud network field, select the network to create the master in.

      • In the Subnet field, select the subnet to create the master in.

        For the regional master, specify three subnets in each availability zone.

    9. Under Maintenance window settings:
      • In the Maintenance frequency / Disable field, configure the maintenance window:
        • Disabled: Automatic updates are disabled.
        • Anytime: Maintenance is allowed at any time.
        • Daily: Maintenance is performed in the time interval specified in the Time (UTC) and duration field.
        • On selected days: Maintenance is performed in the time interval specified in the Weekly schedule field.
    10. Under Cluster network settings:
      • Specify the cluster CIDR, which is a range of IP addresses for allocating pod addresses.
      • Specify the service CIDR, which is a range of IP addresses for allocating service addresses.
    11. Click Create cluster.

    Run the command:

    yc managed-kubernetes cluster create \
  --name cluster-from-backup \
  --network-name network \
  --node-service-account-name docker \
  --service-account-name k8s \
  --zone ru-central1-a

    Where:

    • name: The Kubernetes cluster name.
    • network-name: The name of the network.
    • node-service-account-id: The unique ID of the service account for the nodes. Nodes will download the Docker images they require from the registry on behalf of this account.
    • service-account-id: The unique ID of the service account for the resources. The resources that the Kubernetes cluster needs will be created on behalf of this account.
    • zone: Availability zone.

    Execution result:

    done (6m9s)
id: catab1c2f0fmb3d0ef94
folder_id: b1g23ga82bcv0cdeferg
created_at: "2020-10-19T17:41:28Z"
name: cluster-from-backup
status: RUNNING
health: HEALTHY
network_id: enpabc1rfondide2fflu3
master:
  zonal_master:
    zone_id: ru-central1-a
    internal_v4_address: 10.0.0.32
  version: "1.17"
  endpoints:
    internal_v4_endpoint: https://10.0.0.32
  master_auth:
    cluster_ca_certificate: |
      -----BEGIN CERTIFICATE-----
      MIIAbCCCAbCgAwIBAgIBADANBgkqhkiG1w0BAQsFADAVMRMwEQYDVQQDEwdefWJl
      ...
      pi0jAbcDCLzCkfFuNimHejsSvVFN1N2bYYBCBMkhaYDzV5Ypfy/De0fHJ9U=
      -----END CERTIFICATE-----
  version_info:
    current_version: "1.17"
  maintenance_policy:
    auto_upgrade: true
    maintenance_window:
      anytime: {}
ip_allocation_policy:
  cluster_ipv4_cidr_block: 10.112.0.0/16
  node_ipv4_cidr_mask_size: "24
  service_ipv4_cidr_block: 10.96.0.0/16
service_account_id: ajeabcfldeftb1238n99
node_service_account_id: ajea1b023cmndeft7t7j
release_channel: REGULAR

    To create a Kubernetes cluster, use the create method for the Cluster resource.

  2. Create a node group.

    When creating a node group, specify the public IP address so that the nodes have internet access.

    How to create a node group
    1. In management console, select the folder where you want to create your Kubernetes cluster.
    2. In the list of services, select Managed Service for Kubernetes.
    3. Select the Kubernetes cluster to create a node group for.
    4. On the Kubernetes cluster page, go to the Node groups tab.
    5. Click Create node group.
    6. Enter a name and description for the node group.
    7. Specify the Kubernetes version for the nodes.
    8. Specify the number of nodes in the node group.
    9. In the Scalability section:
      • Select the scaling policy type.
      • Specify the number of nodes in the node group.
    10. Under Allow when creating and updating, specify the maximum number of instances that you can exceed and reduce the size of the group by.
    11. Under Computing resources:
    12. Under Storage:
      • Specify the Disk type:
        • HDD: Standard network drive. Network block storage on an HDD.
        • SSD: Fast network drive. Network block storage on an SSD.
      • Specify the disk size.
    13. Under Network settings:
      • In the Public IP field, choose a method for assigning an IP address:
        • Auto: Assign a random IP address from the Yandex.Cloud IP pool.
      • Specify how nodes should be distributed across availability zones and networks.
    14. Under Access, specify the information required to access the node:
      • Enter the username in the Login field.
      • In the SSH key field, paste the contents of the public key file.
    15. Under Maintenance window settings:
      • In the Maintenance frequency / Disable field, choose the maintenance window:
        • Disabled: Automatic updates are disabled.
        • Anytime: Maintenance is allowed at any time.
        • Daily: Maintenance is performed during the interval specified in the Time (UTC) and duration field.
        • On selected days: Maintenance is performed during the interval specified in the Schedule by day field.
    16. Click Create node group.

    Run the command:

    yc managed-kubernetes node-group create \
  --cluster-name cluster-from-backup \
  --fixed-size 1 \
  --public-ip

    Where:

    • cluster-name: The name of the Kubernetes cluster where the node group is created.
    • fixed-size: The number of nodes in the group.
    • public-ip: Flag specified if the node group needs a public IP address.

    Execution result:

    done (2m31s)
id: catabcio1iq2sdefial3
cluster_id: catab1c2f0fmb3d0ef94
created_at: "2020-10-19T17:43:54Z"
status: RUNNING
node_template:
  platform_id: standard-v2
  resources_spec:
    memory: "4294967296"
    cores: "2"
    core_fraction: "100"
  boot_disk_spec:
    disk_type_id: network-hdd
    disk_size: "103079215104"
  v4_address_spec:
    one_to_one_nat_spec:
      ip_version: IPV4
  scheduling_policy: {}
scale_policy:
  fixed_scale:
    size: "1"
allocation_policy:
  locations:
  - zone_id: ru-central1-a
    subnet_id: e1bm23abcs81edef6dqg
deploy_policy:
  max_expansion: "3"
instance_group_id: cl1ca2bch3lde1fi2tk7
node_version: "1.17"
version_info:
  current_version: "1.17"
maintenance_policy:
  auto_upgrade: true
  auto_repair: true
  maintenance_window:
    anytime: {}

    To create a node group, use the create method for the NodeGroup resource.

  3. Install Velero in the Kubernetes cluster to restore the data in:

    velero install \
  --backup-location-config s3Url=https://storage.yandexcloud.net,region=ru-central1 \
  --bucket velero-backup \
  --plugins velero/velero-plugin-for-aws:v1.0.0 \
  --provider aws \
  --secret-file ./credentials \
  --use-restic \
  --use-volume-snapshots false

    Where:

    • backup-location-config: Configuration of the backup storage address. URL of Object Storage storage and all availability zones.
    • bucket: Name of the backup storage bucket.
    • plugins: Image plugin to enable compatibility with AWS API.
    • provider: Name of the Object Storage object storage provider.
    • secret-file: Name of the file with a static access key and the path to it.
    • use-restic: Use restic to back up Persistent Volumes.
    • use-volume-snapshots: Option to create storage snapshots. Possible values: false — don't create, true — create. Default value: true.

    Execution result:

    CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: already exists, proceeding
CustomResourceDefinition/backups.velero.io: created
...
DaemonSet/restic: already exists, proceeding
DaemonSet/restic: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

  4. Make sure that the data backup is displayed in the created Kubernetes cluster:

    velero backup get

    Execution result:

    NAME        STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
my-backup   Completed   0        0          2020-10-19 17:13:25 +0300 MSK   29d       default            <none>

  5. Restore data from the backup:

    velero restore create create --exclude-namespaces velero --from-backup my-backup

    Where:

    • exclude-namespaces: Parameter that allows users not to restore objects from the velero namespace.
    • from-backup: Name of the bucket where the backup is stored.

    Execution result:

    Restore request "create" submitted successfully.
Run `velero restore describe create` or `velero restore logs create` for more details.

  6. Wait for the backup restoration to complete. The STATUS will be Completed.

    velero get restore

    Execution result:

    NAME     BACKUP    STATUS      STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
create   my-backup   Completed   2020-10-20 14:04:55 +0300 MSK   2020-10-20 14:05:22 +0300 MSK   0        23         2020-10-20 14:04:55 +0300 MSK   <none>
In this article: