Get started
Yandex Data Proc

SubclusterService

Written by

A set of methods for managing Data Proc subclusters.

Call Description
Get Returns the specified subcluster.
List Retrieves a list of subclusters in the specified cluster.
Create Creates a subcluster in the specified cluster.
Update Updates the specified subcluster.
Delete Deletes the specified subcluster.

Calls SubclusterService

Get

Returns the specified subcluster.
To get the list of all available subclusters, make a SubclusterService.List request.

rpc Get (GetSubclusterRequest) returns (Subcluster)

GetSubclusterRequest

Field Description
cluster_id string
Required. ID of the Data Proc cluster that the subcluster belongs to. The maximum string length in characters is 50.
subcluster_id string
Required. ID of the subcluster to return.
To get a subcluster ID make a SubclusterService.List request. The maximum string length in characters is 50.

Subcluster

Field Description
id string
ID of the subcluster. Generated at creation time.
cluster_id string
ID of the Data Proc cluster that the subcluster belongs to.
created_at google.protobuf.Timestamp
Creation timestamp.
name string
Name of the subcluster. The name is unique within the cluster. The string length in characters must be 1-63.
role enum Role
Role that is fulfilled by hosts of the subcluster.
  • MASTERNODE: The subcluster fulfills the master role.
    Master can run the following services, depending on the requested components:
    • HDFS: Namenode, Secondary Namenode
    • YARN: ResourceManager, Timeline Server
    • HBase Master
    • Hive: Server, Metastore, HCatalog
    • Spark History Server
    • Zeppelin
    • ZooKeeper
  • DATANODE: The subcluster is a DATANODE in a Data Proc cluster.
    DATANODE can run the following services, depending on the requested components:
    • HDFS DataNode
    • YARN NodeManager
    • HBase RegionServer
    • Spark libraries
  • COMPUTENODE: The subcluster is a COMPUTENODE in a Data Proc cluster.
    COMPUTENODE can run the following services, depending on the requested components:
    • YARN NodeManager
    • Spark libraries
resources Resources
Resources allocated for each host in the subcluster.
subnet_id string
ID of the VPC subnet used for hosts in the subcluster.
hosts_count int64
Number of hosts in the subcluster.
assign_public_ip bool
Assign public ip addresses for all hosts in subcluter.
autoscaling_config AutoscalingConfig
Configuration for instance group based subclusters
instance_group_id string
ID of Compute Instance Group for autoscaling subclusters

Resources

Field Description
resource_preset_id string
ID of the resource preset for computational resources available to a host (CPU, memory etc.). All available presets are listed in the documentation.
disk_type_id string
Type of the storage environment for the host. Possible values:
  • network-hdd - network HDD drive,
  • network-ssd - network SSD drive.
disk_size int64
Volume of the storage available to a host, in bytes.

AutoscalingConfig

Field Description
max_hosts_count int64
Upper limit for total instance subcluster count. Acceptable values are 1 to 100, inclusive.
preemptible bool
Preemptible instances are stopped at least once every 24 hours, and can be stopped at any time if their resources are needed by Compute. For more information, see Preemptible Virtual Machines.
measurement_duration google.protobuf.Duration
Required. Time in seconds allotted for averaging metrics. Acceptable values are 1m to 10m, inclusive.
warmup_duration google.protobuf.Duration
The warmup time of the instance in seconds. During this time, traffic is sent to the instance, but instance metrics are not collected. The maximum value is 10m.
stabilization_duration google.protobuf.Duration
Minimum amount of time in seconds allotted for monitoring before Instance Groups can reduce the number of instances in the group. During this time, the group size doesn't decrease, even if the new metric values indicate that it should. Acceptable values are 1m to 30m, inclusive.
cpu_utilization_target double
Defines an autoscaling rule based on the average CPU utilization of the instance group. Acceptable values are 10 to 100, inclusive.
decommission_timeout int64
Timeout to gracefully decommission nodes during downscaling. In seconds. Default value: 120 Acceptable values are 0 to 86400, inclusive.

List

Retrieves a list of subclusters in the specified cluster.

rpc List (ListSubclustersRequest) returns (ListSubclustersResponse)

ListSubclustersRequest

Field Description
cluster_id string
Required. ID of the Data Proc cluster to list subclusters in. The maximum string length in characters is 50.
page_size int64
The maximum number of results per page to return. If the number of available results is larger than page_size, the service returns a ListSubclustersResponse.next_page_token that can be used to get the next page of results in subsequent list requests. Default value: 100. The maximum value is 1000.
page_token string
Page token. To get the next page of results, set page_token to the ListSubclustersResponse.next_page_token returned by a previous list request. The maximum string length in characters is 100.
filter string
A filter expression that filters subclusters listed in the response.
The expression must specify:
  1. The field name. Currently you can use filtering only on Subcluster.name field.
  2. An = operator.
  3. The value in double quotes ("). Must be 3-63 characters long and match the regular expression [a-z][-a-z0-9]{1,61}[a-z0-9].
Example of a filter: name=dataproc123_subcluster456. The maximum string length in characters is 1000.

ListSubclustersResponse

Field Description
subclusters[] Subcluster
List of subclusters in the specified cluster.
next_page_token string
Token for getting the next page of the list. If the number of results is greater than the specified ListSubclustersRequest.page_size, use next_page_token as the value for the ListSubclustersRequest.page_token parameter in the next list request.
Each subsequent page will have its own next_page_token to continue paging through the results.

Subcluster

Field Description
id string
ID of the subcluster. Generated at creation time.
cluster_id string
ID of the Data Proc cluster that the subcluster belongs to.
created_at google.protobuf.Timestamp
Creation timestamp.
name string
Name of the subcluster. The name is unique within the cluster. The string length in characters must be 1-63.
role enum Role
Role that is fulfilled by hosts of the subcluster.
  • MASTERNODE: The subcluster fulfills the master role.
    Master can run the following services, depending on the requested components:
    • HDFS: Namenode, Secondary Namenode
    • YARN: ResourceManager, Timeline Server
    • HBase Master
    • Hive: Server, Metastore, HCatalog
    • Spark History Server
    • Zeppelin
    • ZooKeeper
  • DATANODE: The subcluster is a DATANODE in a Data Proc cluster.
    DATANODE can run the following services, depending on the requested components:
    • HDFS DataNode
    • YARN NodeManager
    • HBase RegionServer
    • Spark libraries
  • COMPUTENODE: The subcluster is a COMPUTENODE in a Data Proc cluster.
    COMPUTENODE can run the following services, depending on the requested components:
    • YARN NodeManager
    • Spark libraries
resources Resources
Resources allocated for each host in the subcluster.
subnet_id string
ID of the VPC subnet used for hosts in the subcluster.
hosts_count int64
Number of hosts in the subcluster.
assign_public_ip bool
Assign public ip addresses for all hosts in subcluter.
autoscaling_config AutoscalingConfig
Configuration for instance group based subclusters
instance_group_id string
ID of Compute Instance Group for autoscaling subclusters

Resources

Field Description
resource_preset_id string
ID of the resource preset for computational resources available to a host (CPU, memory etc.). All available presets are listed in the documentation.
disk_type_id string
Type of the storage environment for the host. Possible values:
  • network-hdd - network HDD drive,
  • network-ssd - network SSD drive.
disk_size int64
Volume of the storage available to a host, in bytes.

AutoscalingConfig

Field Description
max_hosts_count int64
Upper limit for total instance subcluster count. Acceptable values are 1 to 100, inclusive.
preemptible bool
Preemptible instances are stopped at least once every 24 hours, and can be stopped at any time if their resources are needed by Compute. For more information, see Preemptible Virtual Machines.
measurement_duration google.protobuf.Duration
Required. Time in seconds allotted for averaging metrics. Acceptable values are 1m to 10m, inclusive.
warmup_duration google.protobuf.Duration
The warmup time of the instance in seconds. During this time, traffic is sent to the instance, but instance metrics are not collected. The maximum value is 10m.
stabilization_duration google.protobuf.Duration
Minimum amount of time in seconds allotted for monitoring before Instance Groups can reduce the number of instances in the group. During this time, the group size doesn't decrease, even if the new metric values indicate that it should. Acceptable values are 1m to 30m, inclusive.
cpu_utilization_target double
Defines an autoscaling rule based on the average CPU utilization of the instance group. Acceptable values are 10 to 100, inclusive.
decommission_timeout int64
Timeout to gracefully decommission nodes during downscaling. In seconds. Default value: 120 Acceptable values are 0 to 86400, inclusive.

Create

Creates a subcluster in the specified cluster.

rpc Create (CreateSubclusterRequest) returns (operation.Operation)

Metadata and response of Operation:

    Operation.metadata:CreateSubclusterMetadata

    Operation.response:Subcluster

CreateSubclusterRequest

Field Description
cluster_id string
Required. ID of the Data Proc cluster to create a subcluster in.
To get a cluster ID, make a ClusterService.List request. The maximum string length in characters is 50.
name string
Name of the subcluster. The name must be unique within the cluster. The name can't be changed when the subcluster is created. Value must match the regular expression |[a-z][-a-z0-9]{1,61}[a-z0-9].
role enum Role
Required. Role that is fulfilled by hosts of the subcluster.
  • MASTERNODE: The subcluster fulfills the master role.
    Master can run the following services, depending on the requested components:
    • HDFS: Namenode, Secondary Namenode
    • YARN: ResourceManager, Timeline Server
    • HBase Master
    • Hive: Server, Metastore, HCatalog
    • Spark History Server
    • Zeppelin
    • ZooKeeper
  • DATANODE: The subcluster is a DATANODE in a Data Proc cluster.
    DATANODE can run the following services, depending on the requested components:
    • HDFS DataNode
    • YARN NodeManager
    • HBase RegionServer
    • Spark libraries
  • COMPUTENODE: The subcluster is a COMPUTENODE in a Data Proc cluster.
    COMPUTENODE can run the following services, depending on the requested components:
    • YARN NodeManager
    • Spark libraries
resources Resources
Required. Resources allocated for each host in the subcluster.
subnet_id string
Required. ID of the VPC subnet used for hosts in the subcluster. The maximum string length in characters is 50.
hosts_count int64
Required. Number of hosts in the subcluster. The minimum value is 1.
autoscaling_config AutoscalingConfig
Configuration for instance group based subclusters

Resources

Field Description
resource_preset_id string
ID of the resource preset for computational resources available to a host (CPU, memory etc.). All available presets are listed in the documentation.
disk_type_id string
Type of the storage environment for the host. Possible values:
  • network-hdd - network HDD drive,
  • network-ssd - network SSD drive.
disk_size int64
Volume of the storage available to a host, in bytes.

AutoscalingConfig

Field Description
max_hosts_count int64
Upper limit for total instance subcluster count. Acceptable values are 1 to 100, inclusive.
preemptible bool
Preemptible instances are stopped at least once every 24 hours, and can be stopped at any time if their resources are needed by Compute. For more information, see Preemptible Virtual Machines.
measurement_duration google.protobuf.Duration
Required. Time in seconds allotted for averaging metrics. Acceptable values are 1m to 10m, inclusive.
warmup_duration google.protobuf.Duration
The warmup time of the instance in seconds. During this time, traffic is sent to the instance, but instance metrics are not collected. The maximum value is 10m.
stabilization_duration google.protobuf.Duration
Minimum amount of time in seconds allotted for monitoring before Instance Groups can reduce the number of instances in the group. During this time, the group size doesn't decrease, even if the new metric values indicate that it should. Acceptable values are 1m to 30m, inclusive.
cpu_utilization_target double
Defines an autoscaling rule based on the average CPU utilization of the instance group. Acceptable values are 10 to 100, inclusive.
decommission_timeout int64
Timeout to gracefully decommission nodes during downscaling. In seconds. Default value: 120 Acceptable values are 0 to 86400, inclusive.

Operation

Field Description
id string
ID of the operation.
description string
Description of the operation. 0-256 characters long.
created_at google.protobuf.Timestamp
Creation timestamp.
created_by string
ID of the user or service account who initiated the operation.
modified_at google.protobuf.Timestamp
The time when the Operation resource was last modified.
done bool
If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.
metadata google.protobuf.Any<CreateSubclusterMetadata>
Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.
result oneof: error or response
The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true, exactly one of error or response is set.
  error google.rpc.Status
The error result of the operation in case of failure or cancellation.
  response google.protobuf.Any<Subcluster>
if operation finished successfully.

CreateSubclusterMetadata

Field Description
cluster_id string
ID of the cluster that the subcluster is being added to. The maximum string length in characters is 50.
subcluster_id string
ID of the subcluster that is being created. The maximum string length in characters is 50.

Subcluster

Field Description
id string
ID of the subcluster. Generated at creation time.
cluster_id string
ID of the Data Proc cluster that the subcluster belongs to.
created_at google.protobuf.Timestamp
Creation timestamp.
name string
Name of the subcluster. The name is unique within the cluster. The string length in characters must be 1-63.
role enum Role
Role that is fulfilled by hosts of the subcluster.
  • MASTERNODE: The subcluster fulfills the master role.
    Master can run the following services, depending on the requested components:
    • HDFS: Namenode, Secondary Namenode
    • YARN: ResourceManager, Timeline Server
    • HBase Master
    • Hive: Server, Metastore, HCatalog
    • Spark History Server
    • Zeppelin
    • ZooKeeper
  • DATANODE: The subcluster is a DATANODE in a Data Proc cluster.
    DATANODE can run the following services, depending on the requested components:
    • HDFS DataNode
    • YARN NodeManager
    • HBase RegionServer
    • Spark libraries
  • COMPUTENODE: The subcluster is a COMPUTENODE in a Data Proc cluster.
    COMPUTENODE can run the following services, depending on the requested components:
    • YARN NodeManager
    • Spark libraries
resources Resources
Resources allocated for each host in the subcluster.
subnet_id string
ID of the VPC subnet used for hosts in the subcluster.
hosts_count int64
Number of hosts in the subcluster.
assign_public_ip bool
Assign public ip addresses for all hosts in subcluter.
autoscaling_config AutoscalingConfig
Configuration for instance group based subclusters
instance_group_id string
ID of Compute Instance Group for autoscaling subclusters

Update

Updates the specified subcluster.

rpc Update (UpdateSubclusterRequest) returns (operation.Operation)

Metadata and response of Operation:

    Operation.metadata:UpdateSubclusterMetadata

    Operation.response:Subcluster

UpdateSubclusterRequest

Field Description
cluster_id string
Required. ID of the cluster to update a subcluster in.
To get a cluster ID, make a ClusterService.List request. The maximum string length in characters is 50.
subcluster_id string
Required. ID of the subcluster to update.
To get a subcluster ID, make a SubclusterService.List request. The maximum string length in characters is 50.
update_mask google.protobuf.FieldMask
Field mask that specifies which attributes of the subcluster should be updated.
resources Resources
New configuration of resources that should be allocated for each host in the subcluster.
name string
New name for the subcluster. The name must be unique within the cluster. Value must match the regular expression |[a-z][-a-z0-9]{1,61}[a-z0-9].
hosts_count int64
Required. New number of hosts in the subcluster. The minimum value is 1.
decommission_timeout int64
Timeout to gracefully decommission nodes. In seconds. Default value: 0 Acceptable values are 0 to 86400, inclusive.
autoscaling_config AutoscalingConfig
Configuration for instance group based subclusters

Resources

Field Description
resource_preset_id string
ID of the resource preset for computational resources available to a host (CPU, memory etc.). All available presets are listed in the documentation.
disk_type_id string
Type of the storage environment for the host. Possible values:
  • network-hdd - network HDD drive,
  • network-ssd - network SSD drive.
disk_size int64
Volume of the storage available to a host, in bytes.

AutoscalingConfig

Field Description
max_hosts_count int64
Upper limit for total instance subcluster count. Acceptable values are 1 to 100, inclusive.
preemptible bool
Preemptible instances are stopped at least once every 24 hours, and can be stopped at any time if their resources are needed by Compute. For more information, see Preemptible Virtual Machines.
measurement_duration google.protobuf.Duration
Required. Time in seconds allotted for averaging metrics. Acceptable values are 1m to 10m, inclusive.
warmup_duration google.protobuf.Duration
The warmup time of the instance in seconds. During this time, traffic is sent to the instance, but instance metrics are not collected. The maximum value is 10m.
stabilization_duration google.protobuf.Duration
Minimum amount of time in seconds allotted for monitoring before Instance Groups can reduce the number of instances in the group. During this time, the group size doesn't decrease, even if the new metric values indicate that it should. Acceptable values are 1m to 30m, inclusive.
cpu_utilization_target double
Defines an autoscaling rule based on the average CPU utilization of the instance group. Acceptable values are 10 to 100, inclusive.
decommission_timeout int64
Timeout to gracefully decommission nodes during downscaling. In seconds. Default value: 120 Acceptable values are 0 to 86400, inclusive.

Operation

Field Description
id string
ID of the operation.
description string
Description of the operation. 0-256 characters long.
created_at google.protobuf.Timestamp
Creation timestamp.
created_by string
ID of the user or service account who initiated the operation.
modified_at google.protobuf.Timestamp
The time when the Operation resource was last modified.
done bool
If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.
metadata google.protobuf.Any<UpdateSubclusterMetadata>
Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.
result oneof: error or response
The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true, exactly one of error or response is set.
  error google.rpc.Status
The error result of the operation in case of failure or cancellation.
  response google.protobuf.Any<Subcluster>
if operation finished successfully.

UpdateSubclusterMetadata

Field Description
cluster_id string
ID of the cluster whose subcluster is being updated. The maximum string length in characters is 50.
subcluster_id string
ID of the subcluster that is being updated. The maximum string length in characters is 50.

Subcluster

Field Description
id string
ID of the subcluster. Generated at creation time.
cluster_id string
ID of the Data Proc cluster that the subcluster belongs to.
created_at google.protobuf.Timestamp
Creation timestamp.
name string
Name of the subcluster. The name is unique within the cluster. The string length in characters must be 1-63.
role enum Role
Role that is fulfilled by hosts of the subcluster.
  • MASTERNODE: The subcluster fulfills the master role.
    Master can run the following services, depending on the requested components:
    • HDFS: Namenode, Secondary Namenode
    • YARN: ResourceManager, Timeline Server
    • HBase Master
    • Hive: Server, Metastore, HCatalog
    • Spark History Server
    • Zeppelin
    • ZooKeeper
  • DATANODE: The subcluster is a DATANODE in a Data Proc cluster.
    DATANODE can run the following services, depending on the requested components:
    • HDFS DataNode
    • YARN NodeManager
    • HBase RegionServer
    • Spark libraries
  • COMPUTENODE: The subcluster is a COMPUTENODE in a Data Proc cluster.
    COMPUTENODE can run the following services, depending on the requested components:
    • YARN NodeManager
    • Spark libraries
resources Resources
Resources allocated for each host in the subcluster.
subnet_id string
ID of the VPC subnet used for hosts in the subcluster.
hosts_count int64
Number of hosts in the subcluster.
assign_public_ip bool
Assign public ip addresses for all hosts in subcluter.
autoscaling_config AutoscalingConfig
Configuration for instance group based subclusters
instance_group_id string
ID of Compute Instance Group for autoscaling subclusters

Delete

Deletes the specified subcluster.

rpc Delete (DeleteSubclusterRequest) returns (operation.Operation)

Metadata and response of Operation:

    Operation.metadata:DeleteSubclusterMetadata

    Operation.response:google.protobuf.Empty

DeleteSubclusterRequest

Field Description
cluster_id string
Required. ID of the cluster to remove a subcluster from.
To get a cluster ID, make a ClusterService.List request. The maximum string length in characters is 50.
subcluster_id string
Required. ID of the subcluster to delete. The maximum string length in characters is 50.
decommission_timeout int64
Timeout to gracefully decommission nodes. In seconds. Default value: 0 Acceptable values are 0 to 86400, inclusive.

Operation

Field Description
id string
ID of the operation.
description string
Description of the operation. 0-256 characters long.
created_at google.protobuf.Timestamp
Creation timestamp.
created_by string
ID of the user or service account who initiated the operation.
modified_at google.protobuf.Timestamp
The time when the Operation resource was last modified.
done bool
If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.
metadata google.protobuf.Any<DeleteSubclusterMetadata>
Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.
result oneof: error or response
The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true, exactly one of error or response is set.
  error google.rpc.Status
The error result of the operation in case of failure or cancellation.
  response google.protobuf.Any<google.protobuf.Empty>
if operation finished successfully.

DeleteSubclusterMetadata

Field Description
cluster_id string
ID of the cluster whose subcluster is being deleted. The maximum string length in characters is 50.
subcluster_id string
ID of the subcluster that is being deleted. The maximum string length in characters is 50.
In this article: