Get started
Yandex Data Proc

Connecting to Data Proc clusters

Written by

After creating a Data Proc cluster, you can connect to the host of the main subcluster.

Cluster hosts cannot be assigned a public IP address, so you need to use a VM from the same Yandex.Cloud network to connect to them.

  1. Create a new VM if necessary.
  2. Connect to the VM via SSH.
  3. You should also use SSH to connect to the host of the main subcluster from your VM.

Alert

When you configure security groups, don't change the rules for service traffic. This may cause the cluster to fail.

  1. Add the rules to the intermediate VM security groups:

    • For incoming traffic:

      • Protocol: TCP.
      • Port range: 22.
      • Source type: CIDR.
      • Destination: 0.0.0.0/0.

    • For outgoing traffic:

      • Protocol: TCP.
      • Port range: 22.
      • Source type: CIDR.
      • Purpose: the address range of the subnet where the hosts of the main subcluster are located.

  2. To connect to main subcluster hosts with an intermediate VM, add rules to the main subcluster host security groups that allow incoming traffic via port 22:

    • Protocol: TCP.
    • Port range: 22.
    • Source type: CIDR.
    • Purpose: the address range of the subnet where the hosts of the main subcluster are located.

To use UI Proxy, add rules to the main subcluster host security group that allow incoming traffic via port 443:

  • Protocol: TCP.
  • Port range: 443.
  • Source type: CIDR.
  • Destination: 0.0.0.0/0.

If the connection is performed via an intermediate VM, add rules to the main subcluster host security group that allow the appropriate connections:

  • For incoming traffic:

    • Protocol: TCP.
    • Port range: 443.
    • Source type: CIDR.
    • Destination: 0.0.0.0/0.

  • For outgoing traffic:

    • Protocol: TCP.
    • Port range: 443.
    • Source type: CIDR.
    • Purpose: the address range of the subnet where the hosts of the main subcluster are located.

When using port forwarding, add rules to the intermediate VM security group that allow incoming and outgoing traffic via the required components' ports:

  • Protocol: TCP.

  • Port range: <component_port>.

    Port numbers for Data Proc components are shown in the table:

    Service Port
    HBase Master 16010
    HBase REST 8085
    HDFS Name Node 9870
    Hive Server2 10002
    MapReduce Application History 19888
    Oozie 11000
    Spark History 18080
    YARN Application History 8188
    YARN Resource Manager 8088
    Zeppelin 8890

  • Source type: CIDR.

  • Destination: 0.0.0.0/0.

Note

You can set more detailed rules for security groups, such as to allow traffic in only specific subnets.

Security groups must be configured correctly for all subnets where cluster hosts will be located.

For more information about security groups, see Security groups.

Connecting to a Data Proc host via SSH

To connect to a Data Proc host from your VM, make sure the SSH key that you specified when creating the Data Proc cluster is accessible on it. You can copy the key to the VM or connect to it with an SSH agent.

  1. Run the SSH agent locally:

    $ eval `ssh-agent -s`

  2. Add the required key to the list of those available to the agent:

    $ ssh-add ~/.ssh/example-key

  3. Open an SSH connection to the Data Proc host for the root user, for example:

    $ ssh root@rc1b-dataproc-m-fh4y4nur0i0uqqkz.mdb.yandexcloud.net

root@rc1b-dataproc-m-fh4y4nur0i0uqqkz:~#

  4. Make sure that Hadoop commands are executed, for example:

    ~# hadoop version

Hadoop 2.8.5
Subversion https://github.yandex-team.ru/mdb/bigtop.git -r 78508f2a4b4f3dc8b3d295ccb50a45a4d24e81b5
Compiled by robot-pgaas-ci on 2019-04-16T10:35Z
Compiled with protoc 2.5.0
From source with checksum 9942ca5c745417c14e318835f420733
This command was run using /usr/lib/hadoop/hadoop-common-2.8.5.jar

Connecting to cluster hosts from graphical IDEs

Connections were tested in the following environment:

  • MacOS Big Sur 11.3:
    • JetBrains DataGrip: 2021.1.
    • DBeaver Community: 21.0.5.
  • Windows 10 Pro 21H1 with SQL Server Management Studio 18.9.1.

You can only use graphical IDEs to connect to cluster hosts through an SSL tunnel using an intermediate VM.

To use graphical IDEs, download the certificate and specify the path to it in the connection settings.

  1. Create a data source:

    1. Select File → New → Data Source → Apache Hive.

      Note

      Select the data source depending on the Data Proc component you're connecting to:

      • Hive: Select Apache Hive.
      • HBase: Select Apache Phoenix.
      • Spark: Select Apache Spark.

      The list of settings doesn't change.

    2. Specify the connection parameters on the General tab:

      • Host: FQDN of the cluster master host.
      • Click Download to download the connection driver.

    3. On the SSH/SSL tab:

      1. Configure the parameters of the SSH tunnel and SSL connection to the VM:
        • Select Use SSH tunnel, create an SSH configuration, and specify the parameters:
          • Host: IP address of the VM.
          • User name: VM user's name.
          • Private key file, Passphrase: File with the private key required to connect to the VM and its password.
      2. Click Test Connection to test the connection to the VM from DataGrip.
      3. Click OK to save the configuration.
      4. Enable the Use SSL setting and specify the SSL connection parameters:
        • CA file: Downloaded SSL certificate for the connection.
        • Client key file, Client key password: File with the private key required to connect to the Data Proc cluster and its password.
  2. Click Test Connection to test the connection. If the connection is successful, you'll see the OK connection status and information about the DBMS and driver.
  3. Click OK to save the data source.
  1. Download the SSH key to the VM to connect to the Data Proc cluster.
  2. Create a new DB connection:

    1. In the Database menu, select New connection.

    2. Select the Apache Hive database from the list.

      Note

      Depending on the Data Proc service you're connecting to, you may need to select Apache Phoenix (for HBase) or Apache Spark instead of Apache Hive. The list of settings doesn't change.

    3. Click Next.

    4. Specify the connection parameters on the Main tab:

      • Host: FQDN of the cluster master host.

    5. On the SSH tab:

      1. Enable the Use SSL tunnel setting.
      2. Specify the SSH tunnel configuration parameters:
        • Host/IP: Public IP address of the VM to connect to.
        • Username: Username for connecting to the VM.
        • Authentication method: Public key.
        • Secret key: Path to the file with the private key used for connecting to the VM.
        • Passphrase: Password of the private key.
  3. Click Test Connection ... to test the connection. If the connection is successful, you'll see the connection status and information about the DBMS and driver.
  4. Click Done to save the database connection settings.
In this article: