Updating a Data Proc cluster
After creating a cluster, you can change its basic and advanced settings.
You can disable sending cluster logs to Yandex Cloud Logging. For more information, see Working with logs.
You can also move a Data Proc cluster to a different availability zone. The process depends on the cluster type:
- Migrating a lightweight cluster to a different availability zone.
- Migrating an HDFS cluster to a different availability zone.
To change Data Proc cluster settings:
-
Go to the folder page
and select Data Proc. -
Select the cluster and click Edit cluster in the top panel.
-
To edit the log group that cluster logs are sent to, select a new log group from the list. If necessary, create a new log group.
To enable this functionality, assign the cluster service account the
logging.writer
role. For more information, see the Yandex Cloud Logging documentation. -
To update your cluster component properties, enter the component key and its new value in the Properties field.
-
Change additional cluster settings:
Deletion protection: Manages cluster protection against accidental deletion by a user.
Enabled protection will not prevent a manual connection to the cluster and deletion of data.
-
Click Save changes.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
-
View a description of the update cluster CLI command:
yc dataproc cluster update --help
-
To edit the log group that cluster logs are sent to, provide the log group ID in the
--log-group-id
parameter:yc dataproc cluster update <cluster_name_or_ID> \ --log-group-id=<log_group_ID>
You can request the log group ID with a list of log groups in the folder.
-
To protect a cluster from accidental deletion by a user of your cloud, set
--deletion-protection
totrue
:yc dataproc cluster update <cluster_name_or_ID> \ --deletion-protection=true
Enabled protection will not prevent a manual connection to the cluster and deletion of data.
-
To update the component properties, provide the required property values in the
--property
parameter:yc dataproc cluster update <cluster_name_or_ID> \ --property "<key_1_prefix>:<key_1>=<value>", "<key_2_prefix>:<key_2>=<value>", ...
Warning
Using the
--property
parameter will reset any component properties that were not explicitly provided in the parameter to their defaults. To save the previously updated properties, list them in your request along with the properties you want to update.
You can get the cluster ID and name with a list of clusters in the folder.
-
Open the current Terraform configuration file with an infrastructure plan.
For more information about how to create this file, see Creating clusters.
-
To activate cluster deletion protection and access to Data Proc component web interfaces, update the values in the appropriate fields of the Data Proc cluster description:
resource "yandex_dataproc_cluster" "data_cluster" { ... deletion_protection = true ui_proxy = true ... }
-
Make sure the settings are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
-
For more information about resources you can create using Terraform, see the provider documentation