Setting up computations on Apache Spark™ clusters

To start computing on Apache Spark™ clusters, prepare a DataSphere project. To run computations, create a new Data Proc cluster or use an existing one.

Set up a project to work with the cluster Data Proc

To set up your project:

1. In the management console, open the DataSphere section in the folder where you work with DataSphere projects.

2. Click in the line with the project to set up.
3. In the window that opens, click Edit.
4. In the window that opens, click Additional settings.
5. In the Service account field, select a service account with the appropriate roles.
6. In the Subnet field, specify the subnet where new Data Proc clusters will be created or existing ones will be used.

Create a cluster Data Proc

Before creating a cluster, make sure you have sufficient resources in your cloud. You can check this in the management console under Quotas.

Create a cluster through the notebook interface in DataSphere

DataSphere monitors the lifetime of the cluster you created and automatically deletes it if it was idle for two hours.

To create a cluster using the notebook interface:

1. In the management console, open the DataSphere section in the folder where you work with DataSphere projects.

2. Select the project to create a Data Proc cluster for.
3. In the top panel, click File and select Data Proc Clusters.
4. In the Create new cluster section of the window that opens:
   1. In the Name field, enter the Data Proc cluster name.
   2. In the Size list, select the Data Proc cluster configuration.
5. Click Create.

Information about the status of the created cluster will be displayed in the same window.

Cluster statuses Data Proc

Once created, a cluster can have the following statuses:

• STARTING: The cluster is being created.
• UP: The cluster is created and ready to run calculations.
• DOWN: Problems creating the cluster.

Create a cluster in Data Proc

You can manage the life cycle of a cluster that you created manually. To ensure correct integration, you need to create a cluster with the following parameters:

• Version: 1.3 and higher.
• Enabled services: LIVY, SPARK, YARN, and HDFS.
• Cluster availability zone: ru-central1-a.

Once the cluster is created, add it to the project settings:

1. In the management console, open the DataSphere section in the folder where you work with DataSphere projects.

2. Click in the line with the project to set up.
3. In the window that opens, click Edit.
4. In the window that opens, click Additional settings.
5. In the Data Proc cluster field, specify the cluster you just created.

Run a computing operation on a Data Proc cluster

To run computations on a cluster created from the notebook interface:

1. In the management console, open the DataSphere section in the folder where you work with DataSphere projects.

2. Click in the line with the project to run computations in.

3. In the cell, insert the code to compute. For example:

   #!spark --cluster <cluster name>
   import random
   
   def inside(p):
       x, y = random.random(), random.random()
       return x*x + y*y < 1
   
   NUM_SAMPLES = 1_000_000
   
   count = sc.parallelize(range(0, NUM_SAMPLES)) \
                .filter(inside).count()
   print("Pi is roughly %f" % (4.0 * count / NUM_SAMPLES))
   

   Where:

   • #!spark --cluster <cluster name> is a mandatory system command to run computations on a cluster. You can view the cluster name in the Data Proc Clusters window, the File menu.

4. Click to run computations.

Wait for the computation to start. While it is in progress, you'll see logs under the cell.

Delete a Data Proc cluster

If you created a cluster using the notebook interface in DataSphere, it will be automatically deleted after two hours of idle time. You can also delete it manually through the notebook interface.

A cluster that you created yourself should be deleted in Data Proc.

Delete a cluster through the notebook interface in DataSphere

To delete a cluster using the notebook interface:

1. In the management console, open the DataSphere section in the folder where you work with DataSphere projects.

2. Select the project to delete the Data Proc cluster from.
3. In the top panel, click File and select Data Proc Clusters.
4. In the window that opens, click Destroy in the cluster line.

Delete a cluster in Data Proc