Get started
Yandex Data Proc

SubclusterService

Written by

A set of methods for managing Data Proc subclusters.

Call Description
Get Returns the specified subcluster.
List Retrieves a list of subclusters in the specified cluster.
Create Creates a subcluster in the specified cluster.
Update Updates the specified subcluster.
Delete Deletes the specified subcluster.

Calls SubclusterService

Get

Returns the specified subcluster.
To get the list of all available subclusters, make a SubclusterService.List request.

rpc Get (GetSubclusterRequest) returns (Subcluster)

GetSubclusterRequest

Field Description
cluster_id string
Required. ID of the Data Proc cluster that the subcluster belongs to. The maximum string length in characters is 50.
subcluster_id string
Required. ID of the subcluster to return.
To get a subcluster ID make a SubclusterService.List request. The maximum string length in characters is 50.

Subcluster

Field Description
id string
ID of the subcluster. Generated at creation time.
cluster_id string
ID of the Data Proc cluster that the subcluster belongs to.
created_at google.protobuf.Timestamp
Creation timestamp.
name string
Name of the subcluster. The name is unique within the cluster. The string length in characters must be 1-63.
role enum Role
Role that is fulfilled by hosts of the subcluster.
  • MASTERNODE:
    • HDFS: Namenode, Secondary Namenode
    • YARN: ResourceManager, Timeline Server
    • HBase Master
    • Hive: Server, Metastore, HCatalog
    • Spark History Server
    • Zeppelin
    • ZooKeeper
  • DATANODE:
    • HDFS DataNode
    • YARN NodeManager
    • HBase RegionServer
    • Spark libraries
  • COMPUTENODE:
    • YARN NodeManager
    • Spark libraries
    resources Resources
    Resources allocated for each host in the subcluster.
    subnet_id string
    ID of the VPC subnet used for hosts in the subcluster.
    hosts_count int64
    Number of hosts in the subcluster.
    autoscaling_config AutoscalingConfig
    Configuration for instance group based subclusters
    instance_group_id string
    ID of Compute Instance Group for autoscaling subclusters

    Resources

    Field Description
    resource_preset_id string
    ID of the resource preset for computational resources available to a host (CPU, memory etc.). All available presets are listed in the documentation.
    disk_type_id string
    • network-hdd - network HDD drive,
    • network-ssd - network SSD drive.
    disk_size int64
    Volume of the storage available to a host, in bytes.

    AutoscalingConfig

    Field Description
    max_hosts_count int64
    Upper limit for total instance subcluster count. Acceptable values are 1 to 100, inclusive.
    preemptible bool
    Preemptible instances are stopped at least once every 24 hours, and can be stopped at any time if their resources are needed by Compute. For more information, see Preemptible Virtual Machines.
    measurement_duration google.protobuf.Duration
    Required. Time in seconds allotted for averaging metrics. Acceptable values are 1m to 10m, inclusive.
    warmup_duration google.protobuf.Duration
    The warmup time of the instance in seconds. During this time, traffic is sent to the instance, but instance metrics are not collected. The maximum value is 10m.
    stabilization_duration google.protobuf.Duration
    Minimum amount of time in seconds allotted for monitoring before Instance Groups can reduce the number of instances in the group. During this time, the group size doesn't decrease, even if the new metric values indicate that it should. Acceptable values are 1m to 30m, inclusive.
    cpu_utilization_target double
    Defines an autoscaling rule based on the average CPU utilization of the instance group. Acceptable values are 10 to 100, inclusive.
    decommission_timeout int64
    Timeout to gracefully decommission nodes during downscaling. In seconds. Default value: 120 Acceptable values are 0 to 86400, inclusive.

    List

    Retrieves a list of subclusters in the specified cluster.

    rpc List (ListSubclustersRequest) returns (ListSubclustersResponse)

    ListSubclustersRequest

    Field Description
    cluster_id string
    Required. ID of the Data Proc cluster to list subclusters in. The maximum string length in characters is 50.
    page_size int64
    The maximum number of results per page to return. If the number of available results is larger than page_size, the service returns a ListSubclustersResponse.next_page_token that can be used to get the next page of results in subsequent list requests. Default value: 100. The maximum value is 1000.
    page_token string
    Page token. To get the next page of results, set page_token to the ListSubclustersResponse.next_page_token returned by a previous list request. The maximum string length in characters is 100.
    filter string
    1. The field name. Currently you can use filtering only on Subcluster.name field.
    2. An operator. Can be either = or != for single values, IN or NOT IN for lists of values.
    3. The value. Must be 3-63 characters long and match the regular expression `^[a-z][-a-z0-9]{1,61}[a-z0-9].
    The maximum string length in characters is 1000.

    ListSubclustersResponse

    Field Description
    subclusters[] Subcluster
    List of subclusters in the specified cluster.
    next_page_token string
    Token for getting the next page of the list. If the number of results is greater than the specified ListSubclustersRequest.page_size, use next_page_token as the value for the ListSubclustersRequest.page_token parameter in the next list request.
    Each subsequent page will have its own next_page_token to continue paging through the results.

    Subcluster

    Field Description
    id string
    ID of the subcluster. Generated at creation time.
    cluster_id string
    ID of the Data Proc cluster that the subcluster belongs to.
    created_at google.protobuf.Timestamp
    Creation timestamp.
    name string
    Name of the subcluster. The name is unique within the cluster. The string length in characters must be 1-63.
    role enum Role
    Role that is fulfilled by hosts of the subcluster.
    • MASTERNODE:
      • HDFS: Namenode, Secondary Namenode
      • YARN: ResourceManager, Timeline Server
      • HBase Master
      • Hive: Server, Metastore, HCatalog
      • Spark History Server
      • Zeppelin
      • ZooKeeper
    • DATANODE:
      • HDFS DataNode
      • YARN NodeManager
      • HBase RegionServer
      • Spark libraries
    • COMPUTENODE:
      • YARN NodeManager
      • Spark libraries
      resources Resources
      Resources allocated for each host in the subcluster.
      subnet_id string
      ID of the VPC subnet used for hosts in the subcluster.
      hosts_count int64
      Number of hosts in the subcluster.
      autoscaling_config AutoscalingConfig
      Configuration for instance group based subclusters
      instance_group_id string
      ID of Compute Instance Group for autoscaling subclusters

      Resources

      Field Description
      resource_preset_id string
      ID of the resource preset for computational resources available to a host (CPU, memory etc.). All available presets are listed in the documentation.
      disk_type_id string
      • network-hdd - network HDD drive,
      • network-ssd - network SSD drive.
      disk_size int64
      Volume of the storage available to a host, in bytes.

      AutoscalingConfig

      Field Description
      max_hosts_count int64
      Upper limit for total instance subcluster count. Acceptable values are 1 to 100, inclusive.
      preemptible bool
      Preemptible instances are stopped at least once every 24 hours, and can be stopped at any time if their resources are needed by Compute. For more information, see Preemptible Virtual Machines.
      measurement_duration google.protobuf.Duration
      Required. Time in seconds allotted for averaging metrics. Acceptable values are 1m to 10m, inclusive.
      warmup_duration google.protobuf.Duration
      The warmup time of the instance in seconds. During this time, traffic is sent to the instance, but instance metrics are not collected. The maximum value is 10m.
      stabilization_duration google.protobuf.Duration
      Minimum amount of time in seconds allotted for monitoring before Instance Groups can reduce the number of instances in the group. During this time, the group size doesn't decrease, even if the new metric values indicate that it should. Acceptable values are 1m to 30m, inclusive.
      cpu_utilization_target double
      Defines an autoscaling rule based on the average CPU utilization of the instance group. Acceptable values are 10 to 100, inclusive.
      decommission_timeout int64
      Timeout to gracefully decommission nodes during downscaling. In seconds. Default value: 120 Acceptable values are 0 to 86400, inclusive.

      Create

      Creates a subcluster in the specified cluster.

      rpc Create (CreateSubclusterRequest) returns (operation.Operation)

      Metadata and response of Operation:

          Operation.metadata:CreateSubclusterMetadata

          Operation.response:Subcluster

      CreateSubclusterRequest

      Field Description
      cluster_id string
      Required. ID of the Data Proc cluster to create a subcluster in.
      To get a cluster ID, make a ClusterService.List request. The maximum string length in characters is 50.
      name string
      Name of the subcluster. The name must be unique within the cluster. The name can't be changed when the subcluster is created. Value must match the regular expression `
      role enum Role
      Required. Role that is fulfilled by hosts of the subcluster.
      • MASTERNODE:
        • HDFS: Namenode, Secondary Namenode
        • YARN: ResourceManager, Timeline Server
        • HBase Master
        • Hive: Server, Metastore, HCatalog
        • Spark History Server
        • Zeppelin
        • ZooKeeper
      • DATANODE:
        • HDFS DataNode
        • YARN NodeManager
        • HBase RegionServer
        • Spark libraries
      • COMPUTENODE:
        • YARN NodeManager
        • Spark libraries
        resources Resources
        Required. Resources allocated for each host in the subcluster.
        subnet_id string
        Required. ID of the VPC subnet used for hosts in the subcluster. The maximum string length in characters is 50.
        hosts_count int64
        Required. Number of hosts in the subcluster. The minimum value is 1.
        autoscaling_config AutoscalingConfig
        Configuration for instance group based subclusters

        Resources

        Field Description
        resource_preset_id string
        ID of the resource preset for computational resources available to a host (CPU, memory etc.). All available presets are listed in the documentation.
        disk_type_id string
        • network-hdd - network HDD drive,
        • network-ssd - network SSD drive.
        disk_size int64
        Volume of the storage available to a host, in bytes.

        AutoscalingConfig

        Field Description
        max_hosts_count int64
        Upper limit for total instance subcluster count. Acceptable values are 1 to 100, inclusive.
        preemptible bool
        Preemptible instances are stopped at least once every 24 hours, and can be stopped at any time if their resources are needed by Compute. For more information, see Preemptible Virtual Machines.
        measurement_duration google.protobuf.Duration
        Required. Time in seconds allotted for averaging metrics. Acceptable values are 1m to 10m, inclusive.
        warmup_duration google.protobuf.Duration
        The warmup time of the instance in seconds. During this time, traffic is sent to the instance, but instance metrics are not collected. The maximum value is 10m.
        stabilization_duration google.protobuf.Duration
        Minimum amount of time in seconds allotted for monitoring before Instance Groups can reduce the number of instances in the group. During this time, the group size doesn't decrease, even if the new metric values indicate that it should. Acceptable values are 1m to 30m, inclusive.
        cpu_utilization_target double
        Defines an autoscaling rule based on the average CPU utilization of the instance group. Acceptable values are 10 to 100, inclusive.
        decommission_timeout int64
        Timeout to gracefully decommission nodes during downscaling. In seconds. Default value: 120 Acceptable values are 0 to 86400, inclusive.

        Operation

        Field Description
        id string
        ID of the operation.
        description string
        Description of the operation. 0-256 characters long.
        created_at google.protobuf.Timestamp
        Creation timestamp.
        created_by string
        ID of the user or service account who initiated the operation.
        modified_at google.protobuf.Timestamp
        The time when the Operation resource was last modified.
        done bool
        If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.
        metadata google.protobuf.Any<CreateSubclusterMetadata>
        Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.
        result oneof: error or response
        The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true, exactly one of error or response is set.
          error google.rpc.Status
        The error result of the operation in case of failure or cancellation.
          response google.protobuf.Any<Subcluster>
        if operation finished successfully.

        CreateSubclusterMetadata

        Field Description
        cluster_id string
        ID of the cluster that the subcluster is being added to. The maximum string length in characters is 50.
        subcluster_id string
        ID of the subcluster that is being created. The maximum string length in characters is 50.

        Subcluster

        Field Description
        id string
        ID of the subcluster. Generated at creation time.
        cluster_id string
        ID of the Data Proc cluster that the subcluster belongs to.
        created_at google.protobuf.Timestamp
        Creation timestamp.
        name string
        Name of the subcluster. The name is unique within the cluster. The string length in characters must be 1-63.
        role enum Role
        Role that is fulfilled by hosts of the subcluster.
        • MASTERNODE:
          • HDFS: Namenode, Secondary Namenode
          • YARN: ResourceManager, Timeline Server
          • HBase Master
          • Hive: Server, Metastore, HCatalog
          • Spark History Server
          • Zeppelin
          • ZooKeeper
        • DATANODE:
          • HDFS DataNode
          • YARN NodeManager
          • HBase RegionServer
          • Spark libraries
        • COMPUTENODE:
          • YARN NodeManager
          • Spark libraries
          resources Resources
          Resources allocated for each host in the subcluster.
          subnet_id string
          ID of the VPC subnet used for hosts in the subcluster.
          hosts_count int64
          Number of hosts in the subcluster.
          autoscaling_config AutoscalingConfig
          Configuration for instance group based subclusters
          instance_group_id string
          ID of Compute Instance Group for autoscaling subclusters

          Resources

          Field Description
          resource_preset_id string
          ID of the resource preset for computational resources available to a host (CPU, memory etc.). All available presets are listed in the documentation.
          disk_type_id string
          • network-hdd - network HDD drive,
          • network-ssd - network SSD drive.
          disk_size int64
          Volume of the storage available to a host, in bytes.

          AutoscalingConfig

          Field Description
          max_hosts_count int64
          Upper limit for total instance subcluster count. Acceptable values are 1 to 100, inclusive.
          preemptible bool
          Preemptible instances are stopped at least once every 24 hours, and can be stopped at any time if their resources are needed by Compute. For more information, see Preemptible Virtual Machines.
          measurement_duration google.protobuf.Duration
          Required. Time in seconds allotted for averaging metrics. Acceptable values are 1m to 10m, inclusive.
          warmup_duration google.protobuf.Duration
          The warmup time of the instance in seconds. During this time, traffic is sent to the instance, but instance metrics are not collected. The maximum value is 10m.
          stabilization_duration google.protobuf.Duration
          Minimum amount of time in seconds allotted for monitoring before Instance Groups can reduce the number of instances in the group. During this time, the group size doesn't decrease, even if the new metric values indicate that it should. Acceptable values are 1m to 30m, inclusive.
          cpu_utilization_target double
          Defines an autoscaling rule based on the average CPU utilization of the instance group. Acceptable values are 10 to 100, inclusive.
          decommission_timeout int64
          Timeout to gracefully decommission nodes during downscaling. In seconds. Default value: 120 Acceptable values are 0 to 86400, inclusive.

          Update

          Updates the specified subcluster.

          rpc Update (UpdateSubclusterRequest) returns (operation.Operation)

          Metadata and response of Operation:

              Operation.metadata:UpdateSubclusterMetadata

              Operation.response:Subcluster

          UpdateSubclusterRequest

          Field Description
          cluster_id string
          Required. ID of the cluster to update a subcluster in.
          To get a cluster ID, make a ClusterService.List request. The maximum string length in characters is 50.
          subcluster_id string
          Required. ID of the subcluster to update.
          To get a subcluster ID, make a SubclusterService.List request. The maximum string length in characters is 50.
          update_mask google.protobuf.FieldMask
          Field mask that specifies which attributes of the subcluster should be updated.
          resources Resources
          New configuration of resources that should be allocated for each host in the subcluster.
          name string
          New name for the subcluster. The name must be unique within the cluster. Value must match the regular expression `
          hosts_count int64
          Required. New number of hosts in the subcluster. The minimum value is 1.
          decommission_timeout int64
          Timeout to gracefully decommission nodes. In seconds. Default value: 0 Acceptable values are 0 to 86400, inclusive.
          autoscaling_config AutoscalingConfig
          Configuration for instance group based subclusters

          Resources

          Field Description
          resource_preset_id string
          ID of the resource preset for computational resources available to a host (CPU, memory etc.). All available presets are listed in the documentation.
          disk_type_id string
          • network-hdd - network HDD drive,
          • network-ssd - network SSD drive.
          disk_size int64
          Volume of the storage available to a host, in bytes.

          AutoscalingConfig

          Field Description
          max_hosts_count int64
          Upper limit for total instance subcluster count. Acceptable values are 1 to 100, inclusive.
          preemptible bool
          Preemptible instances are stopped at least once every 24 hours, and can be stopped at any time if their resources are needed by Compute. For more information, see Preemptible Virtual Machines.
          measurement_duration google.protobuf.Duration
          Required. Time in seconds allotted for averaging metrics. Acceptable values are 1m to 10m, inclusive.
          warmup_duration google.protobuf.Duration
          The warmup time of the instance in seconds. During this time, traffic is sent to the instance, but instance metrics are not collected. The maximum value is 10m.
          stabilization_duration google.protobuf.Duration
          Minimum amount of time in seconds allotted for monitoring before Instance Groups can reduce the number of instances in the group. During this time, the group size doesn't decrease, even if the new metric values indicate that it should. Acceptable values are 1m to 30m, inclusive.
          cpu_utilization_target double
          Defines an autoscaling rule based on the average CPU utilization of the instance group. Acceptable values are 10 to 100, inclusive.
          decommission_timeout int64
          Timeout to gracefully decommission nodes during downscaling. In seconds. Default value: 120 Acceptable values are 0 to 86400, inclusive.

          Operation

          Field Description
          id string
          ID of the operation.
          description string
          Description of the operation. 0-256 characters long.
          created_at google.protobuf.Timestamp
          Creation timestamp.
          created_by string
          ID of the user or service account who initiated the operation.
          modified_at google.protobuf.Timestamp
          The time when the Operation resource was last modified.
          done bool
          If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.
          metadata google.protobuf.Any<UpdateSubclusterMetadata>
          Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.
          result oneof: error or response
          The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true, exactly one of error or response is set.
            error google.rpc.Status
          The error result of the operation in case of failure or cancellation.
            response google.protobuf.Any<Subcluster>
          if operation finished successfully.

          UpdateSubclusterMetadata

          Field Description
          cluster_id string
          ID of the cluster whose subcluster is being updated. The maximum string length in characters is 50.
          subcluster_id string
          ID of the subcluster that is being updated. The maximum string length in characters is 50.

          Subcluster

          Field Description
          id string
          ID of the subcluster. Generated at creation time.
          cluster_id string
          ID of the Data Proc cluster that the subcluster belongs to.
          created_at google.protobuf.Timestamp
          Creation timestamp.
          name string
          Name of the subcluster. The name is unique within the cluster. The string length in characters must be 1-63.
          role enum Role
          Role that is fulfilled by hosts of the subcluster.
          • MASTERNODE:
            • HDFS: Namenode, Secondary Namenode
            • YARN: ResourceManager, Timeline Server
            • HBase Master
            • Hive: Server, Metastore, HCatalog
            • Spark History Server
            • Zeppelin
            • ZooKeeper
          • DATANODE:
            • HDFS DataNode
            • YARN NodeManager
            • HBase RegionServer
            • Spark libraries
          • COMPUTENODE:
            • YARN NodeManager
            • Spark libraries
            resources Resources
            Resources allocated for each host in the subcluster.
            subnet_id string
            ID of the VPC subnet used for hosts in the subcluster.
            hosts_count int64
            Number of hosts in the subcluster.
            autoscaling_config AutoscalingConfig
            Configuration for instance group based subclusters
            instance_group_id string
            ID of Compute Instance Group for autoscaling subclusters

            Resources

            Field Description
            resource_preset_id string
            ID of the resource preset for computational resources available to a host (CPU, memory etc.). All available presets are listed in the documentation.
            disk_type_id string
            • network-hdd - network HDD drive,
            • network-ssd - network SSD drive.
            disk_size int64
            Volume of the storage available to a host, in bytes.

            AutoscalingConfig

            Field Description
            max_hosts_count int64
            Upper limit for total instance subcluster count. Acceptable values are 1 to 100, inclusive.
            preemptible bool
            Preemptible instances are stopped at least once every 24 hours, and can be stopped at any time if their resources are needed by Compute. For more information, see Preemptible Virtual Machines.
            measurement_duration google.protobuf.Duration
            Required. Time in seconds allotted for averaging metrics. Acceptable values are 1m to 10m, inclusive.
            warmup_duration google.protobuf.Duration
            The warmup time of the instance in seconds. During this time, traffic is sent to the instance, but instance metrics are not collected. The maximum value is 10m.
            stabilization_duration google.protobuf.Duration
            Minimum amount of time in seconds allotted for monitoring before Instance Groups can reduce the number of instances in the group. During this time, the group size doesn't decrease, even if the new metric values indicate that it should. Acceptable values are 1m to 30m, inclusive.
            cpu_utilization_target double
            Defines an autoscaling rule based on the average CPU utilization of the instance group. Acceptable values are 10 to 100, inclusive.
            decommission_timeout int64
            Timeout to gracefully decommission nodes during downscaling. In seconds. Default value: 120 Acceptable values are 0 to 86400, inclusive.

            Delete

            Deletes the specified subcluster.

            rpc Delete (DeleteSubclusterRequest) returns (operation.Operation)

            Metadata and response of Operation:

                Operation.metadata:DeleteSubclusterMetadata

                Operation.response:google.protobuf.Empty

            DeleteSubclusterRequest

            Field Description
            cluster_id string
            Required. ID of the cluster to remove a subcluster from.
            To get a cluster ID, make a ClusterService.List request. The maximum string length in characters is 50.
            subcluster_id string
            Required. ID of the subcluster to delete. The maximum string length in characters is 50.
            decommission_timeout int64
            Timeout to gracefully decommission nodes. In seconds. Default value: 0 Acceptable values are 0 to 86400, inclusive.

            Operation

            Field Description
            id string
            ID of the operation.
            description string
            Description of the operation. 0-256 characters long.
            created_at google.protobuf.Timestamp
            Creation timestamp.
            created_by string
            ID of the user or service account who initiated the operation.
            modified_at google.protobuf.Timestamp
            The time when the Operation resource was last modified.
            done bool
            If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.
            metadata google.protobuf.Any<DeleteSubclusterMetadata>
            Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.
            result oneof: error or response
            The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true, exactly one of error or response is set.
              error google.rpc.Status
            The error result of the operation in case of failure or cancellation.
              response google.protobuf.Any<google.protobuf.Empty>
            if operation finished successfully.

            DeleteSubclusterMetadata

            Field Description
            cluster_id string
            ID of the cluster whose subcluster is being deleted. The maximum string length in characters is 50.
            subcluster_id string
            ID of the subcluster that is being deleted. The maximum string length in characters is 50.
            In this article: