Get started
Language / Region
© 2021 Yandex.Cloud LLC
Yandex Data Proc

Managing PySpark jobs

Written by

Create a job

  1. Go to the folder page and select Data Proc.

  2. Click on the name of the cluster and open the Jobs tab.

  3. Click Submit job.

  4. (Optional) Enter a name for the job.

  5. In the Job type field, select PySpark.

  6. In the Main python file field, specify the path to the main PY application file in the following format:

    File Path format
    Instance file system file:///<file path>
    Distributed cluster file system hdfs:///<file path>
    Object Storage bucket s3a://bucket name>/<file path>
    Internet http://<file path> or https://<file path>

    Archives in standard Linux formats, such as zip, gz, xz, bz2, and others, are supported.

    The cluster service account needs read access to all the files in the bucket. Step-by-step instructions for setting up access to Object Storage are provided in Editing a bucket ACL.

  7. (Optional) Specify the paths to the PY files, if any.

  8. Specify job arguments.

    If an argument, variable, or property is in several space-separated parts, specify each part separately. At the same time, it is important to preserve the order in which you declare arguments, variables, and properties.

    The -mapper mapper.py argument, for instance, must be converted into two arguments: -mapper and mapper.py in that order.

  9. (Optional) Specify the paths to the JAR files, if any.

  10. (Optional) Configure advanced settings:

    • Specify paths to the necessary files and archives.
    • Specify job properties.

  11. Click Submit job.

If you don't have the Yandex.Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a job:

  1. View a description of the CLI create command for PySpark jobs:

    yc dataproc job create-pyspark --help

  2. Create a job (the example doesn't show all the available parameters):

    yc dataproc job create-pyspark \
   --cluster-name <cluster name> \
   --name <job name> \
   --main-python-file-uri <path to main py application file> \
   --python-file-uris <paths to additional py files> \
   --jar-file-uris <paths to jar files> \
   --archive-uris <paths to archives> \
   --properties <job properties> \
   --args <arguments passed to job>

    Pass in the paths to the files required for the job in the following format:

    File Path format
    Instance file system file:///<file path>
    Distributed cluster file system hdfs:///<file path>
    Object Storage bucket s3a://bucket name>/<file path>
    Internet http://<file path> or https://<file path>

    Archives in standard Linux formats, such as zip, gz, xz, bz2, and others, are supported.

    The cluster service account needs read access to all the files in the bucket. Step-by-step instructions for setting up access to Object Storage are provided in Editing a bucket ACL.

You can query the cluster ID and name with a list of clusters in the folder.

Use the create API method and pass the following in the call:

  • The cluster ID in the clusterId parameter.
  • The job name in the name parameter.
  • The job properties in the pysparkJob parameter.

You can get the cluster ID with a list of clusters in the folder.

Get a list of jobs

  1. Go to the folder page and select Data Proc.
  2. Click on the name of the cluster and open the Jobs tab.

If you don't have the Yandex.Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get a list of jobs, run the command:

yc dataproc job list --cluster-name <cluster name>

You can query the cluster ID and name with a list of clusters in the folder.

Use the list API method and pass the cluster ID in the clusterId parameter.

You can get the cluster ID with a list of clusters in the folder.

Get general information about the job

  1. Go to the folder page and select Data Proc.
  2. Click on the name of the cluster and open the Jobs tab.
  3. Click on the name of the job.

If you don't have the Yandex.Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get general information about the job, run the command:

yc dataproc job get \
   --cluster-name <cluster name> \
   --name <job name>

You can query the cluster ID and name with a list of clusters in the folder.

Use the get API method and pass the following in the request:

Get job execution logs

  1. Go to the folder page and select Data Proc.
  2. Click on the name of the cluster and open the Jobs tab.
  3. Click on the name of the job.

If you don't have the Yandex.Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get job execution logs, run the command:

yc dataproc job log \
   --cluster-name <cluster name> \
   --name <job name>

You can query the cluster ID and name with a list of clusters in the folder.

Use the listLog API method and pass the following in the request:

Language / Region
© 2021 Yandex.Cloud LLC
In this article: