Managing Data Proc subclusters
In addition to updating the settings of a particular subcluster, you can create new and delete existing subclusters.
Warning
Each cluster may have only 1 MASTERNODE subcluster, which is why you can't create or delete such subclusters. You can't delete DATANODE subclusters, either.
Getting a list of subclusters
-
Go to the folder page and select Data Proc.
-
Click on the name of the desired cluster and then select the Subclusters tab.
Adding a subcluster
The number of hosts in Data Proc clusters is limited quotas by the number of vCPUs and the amount of RAM that VMs in your cloud can use. To view the resources available, go to the Quotas section and find Compute Cloud.
To add a subcluster:
-
In management console, select the required folder.
-
Select Data Proc and the desired cluster.
-
Go to Subclusters.
-
Click Add subcluster.
-
Select the subcluster type. To do this, decide what services should be deployed on the hosts:
- In COMPUTENODE subclusters, you can deploy:
- YARN NodeManager
- Spark libraries
- In DATANODE subclusters, you can deploy:
- HDFS Datanode
- YARN NodeManager
- HBase RegionServer
- Spark libraries
- In COMPUTENODE subclusters, you can deploy:
-
Select the other subcluster settings:
- The number of hosts.
- The host class is the platform and computing resources available to the host.
- Storage size and type.
- The subnet of the network where the cluster is located.
-
Click Add subcluster.
Data Proc runs the create subcluster operation.
Deleting a subcluster
Warning
You can't delete DATANODE subclusters.
To delete a subcluster:
- In management console, select the required folder.
- Select Data Proc and the desired cluster.
- Go to Subclusters.
- Optionally specify the decommissioning timeout.
- Click for the appropriate subcluster and select Delete.
- Confirm the deletion.
Data Proc runs the delete subcluster operation.