Get started
Yandex Data Proc

Creating Data Proc clusters

Written by

Configure security groups

Warning

Security groups must be created and configured before creating a cluster. If the selected security groups don't have the required rules, Yandex.Cloud disables cluster creation.

  1. Create one or more security groups for cluster service traffic.

  2. Add rules:

    • One rule for incoming and outgoing service traffic:

      • Port range: 0-65535.
      • Protocol: Any.
      • Source type: Security group.
      • Purpose: Current security group (Self).

    • A separate rule for outgoing HTTPS traffic:

      • Port range: 443.
      • Protocol: TCP.
      • Source type: CIDR.
      • Destination: 0.0.0.0/0.

      This will allow you to use Object Storage buckets, UI Proxy, and automatic scaling of clusters.

If you plan to use multiple security groups for a cluster, enable all traffic between these groups.

Note

You can set more detailed rules for security groups, such as to allow traffic in only specific subnets.

Security groups must be configured correctly for all subnets that will include cluster hosts.

You can set up security groups for connections to cluster hosts via an intermediate VM after creating a cluster.

Create a cluster

  1. In the management console, select the folder where you want to create a cluster.

  2. Click Create resource and select Data Proc cluster from the drop-down list.

  3. Enter a name for the cluster in the Cluster name field. The cluster name must be unique within the folder.

  4. Select a relevant image version and the components you want to use in the cluster.

    Note

    Note that some components require other components to work. For example, to use Spark, you need YARN.

  5. Enter the public part of your SSH key in the Public key field. For information about how to generate and use SSH keys, see the Yandex Compute Cloud documentation.

  6. Select or create a service account that you want to grant access to the cluster.

  7. Select the availability zone for the cluster.

  8. If necessary, set the properties of Hadoop and its components, for example:

    hdfs:dfs.replication : 2
hdfs:dfs.blocksize : 1073741824
spark:spark.driver.cores : 1

    The available properties are listed in the official documentation for the components:

  9. Select or create a network for the cluster.

  10. Select security groups that have the required permissions.

    Warning

    When creating a cluster, security group settings are verified. If a cluster cannot function with these settings, you get a warning. See above for an example of working settings.

  11. Enable the UI Proxy option to access the web interfaces of the components Data Proc.

  12. Configure subclusters: no more than one main subcluster with a Master host and subclusters for data storage or computing.

    The roles of Compute and Data subcluster are different: you can deploy data storage components on Data subclusters, and data processing components on Compute subclusters. Storage on a Compute subcluster is only used to temporarily store processed files.

  13. For each subcluster, you can configure:

    • The number of hosts.
    • The host class is the platform and computing resources available to the host.
    • Storage size and type.
    • The subnet of the network where the cluster is located.

  14. For Compute subclusters, you can specify the autoscaling parameters.

    Note

    To enable automatic scaling, assign the following roles to the cluster service account:

    • editor
    • dataproc.agent
    1. Under Add subcluster, click Add.
    2. In the Roles field, select COMPUTENODE.
    3. Under Scalability, enable Automatic scaling.
    4. Set autoscaling parameters.
    5. The default metric used for autoscaling is yarn.cluster.containersPending. To enable scaling based on CPU usage, disable the Default scaling option and set the target CPU utilization level.
    6. Click Add.

  15. After you configure all the subclusters you need, click Create cluster.

Data Proc runs the create cluster operation. After the cluster status changes to Running, you can connect to any active subcluster using the specified SSH key.

In this article: