DataSphere API v2, gRPC: ProjectJobService
A set of methods for managing Project Jobs. Do not use these methods manually.
For working with DataSphere Jobs, install DataSphere CLI via pip install datasphere
.
Call | Description |
---|---|
Create | Creates job. |
Execute | Runs job execution. |
Cancel | Cancels running job. |
ReadStdLogs | Returns stream of job logs. |
ReadLogs | Returns stream of job logs. |
DownloadJobFiles | Returns download urls for job files. |
List | Lists jobs. |
Get | Returns job by id. |
Delete | Deletes specified job. |
DeleteData | Delete job data. |
DeleteAllData | Delete all jobs data. |
Calls ProjectJobService
Create
Creates job.
rpc Create (CreateProjectJobRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:CreateProjectJobMetadata
Operation.response:CreateProjectJobResponse
CreateProjectJobRequest
Field | Description |
---|---|
project_id | string ID of the project. |
job_parameters | JobParameters Parameters of the job. |
config | string Config of the job. |
name | string Name of the job. |
desc | string Description of the job. |
data_ttl | google.protobuf.Duration Job data TTL. |
JobParameters
Field | Description |
---|---|
input_files[] | File List of input files. |
output_files[] | FileDesc List of output files descriptions. |
s3_mount_ids[] | string List of DataSphere S3 mount ids. |
dataset_ids[] | string List of DataSphere dataset ids. |
cmd | string Job run command. |
env | Environment Job environment description. |
attach_project_disk | bool Should project disk be attached to VM. |
cloud_instance_type | CloudInstanceType VM specification. |
extended_working_storage | ExtendedWorkingStorage Extended working storage configuration. |
File
Field | Description |
---|---|
desc | FileDesc |
sha256 | string SHA256 of the file. |
size_bytes | int64 File size in bytes. |
FileDesc
Field | Description |
---|---|
path | string Path of the file on filesystem. |
var | string Variable to use in cmd substitution. |
Environment
Field | Description |
---|---|
vars | map<string,string> Environment variables. |
docker_image | oneof: docker_image_resource_id or docker_image_spec |
docker_image_resource_id | string DS docker image id. |
docker_image_spec | DockerImageSpec |
python_env | PythonEnv |
DockerImageSpec
Field | Description |
---|---|
image_url | string Docker image URL. |
username | string Username for container registry. |
password | oneof: password_plain_text or password_ds_secret_name Password for container registry. |
password_plain_text | string Plaintext password. |
password_ds_secret_name | string ID of DataSphere secret containing password. |
PythonEnv
Field | Description |
---|---|
conda_yaml | string Conda YAML. |
local_modules[] | File List of local modules descriptions. |
CloudInstanceType
Field | Description |
---|---|
name | string Name of DataSphere VM configuration. |
ExtendedWorkingStorage
Field | Description |
---|---|
type | enum StorageType |
size_gb | int64 |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
CreateProjectJobMetadata
Field | Description |
---|---|
project_id | string ID of the project. |
job_id | string Job ID. |
CreateProjectJobResponse
Field | Description |
---|---|
job_id | string ID of the job. |
upload_files[] | StorageFile Files to upload with their presigned URLs for upload. |
StorageFile
Field | Description |
---|---|
file | File |
url | string File URL. |
Execute
Runs job execution.
rpc Execute (ExecuteProjectJobRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:ExecuteProjectJobMetadata
Operation.response:ExecuteProjectJobResponse
ExecuteProjectJobRequest
Field | Description |
---|---|
job_id | string ID of the job. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
ExecuteProjectJobMetadata
Field | Description |
---|---|
job | Job Instance of the job. |
Job
Field | Description |
---|---|
id | string ID of the job. |
name | string Name of the job. |
desc | string Description of the job. |
created_at | google.protobuf.Timestamp Create job timestamp. |
started_at | google.protobuf.Timestamp Start job timestamp. |
finished_at | google.protobuf.Timestamp Finish job timestamp. |
status | enum JobStatus Status of the job. |
config | string Config of the job, copied from configuration file. |
created_by_id | string ID of the user who created the job. |
project_id | string ID of the project. |
job_parameters | JobParameters |
data_expires_at | google.protobuf.Timestamp Job data expiration timestamp. |
data_cleared | bool Marks if the job data has been cleared. |
output_files[] | File Output files of the job. |
log_files[] | File Job log files. |
diagnostic_files[] | File Job diagnostics files. |
data_size_bytes | int64 Job total data size. |
JobParameters
Field | Description |
---|---|
input_files[] | File List of input files. |
output_files[] | FileDesc List of output files descriptions. |
s3_mount_ids[] | string List of DataSphere S3 mount ids. |
dataset_ids[] | string List of DataSphere dataset ids. |
cmd | string Job run command. |
env | Environment Job environment description. |
attach_project_disk | bool Should project disk be attached to VM. |
cloud_instance_type | CloudInstanceType VM specification. |
extended_working_storage | ExtendedWorkingStorage Extended working storage configuration. |
FileDesc
Field | Description |
---|---|
path | string Path of the file on filesystem. |
var | string Variable to use in cmd substitution. |
Environment
Field | Description |
---|---|
vars | map<string,string> Environment variables. |
docker_image | oneof: docker_image_resource_id or docker_image_spec |
docker_image_resource_id | string DS docker image id. |
docker_image_spec | DockerImageSpec |
python_env | PythonEnv |
DockerImageSpec
Field | Description |
---|---|
image_url | string Docker image URL. |
username | string Username for container registry. |
password | oneof: password_plain_text or password_ds_secret_name Password for container registry. |
password_plain_text | string Plaintext password. |
password_ds_secret_name | string ID of DataSphere secret containing password. |
PythonEnv
Field | Description |
---|---|
conda_yaml | string Conda YAML. |
local_modules[] | File List of local modules descriptions. |
CloudInstanceType
Field | Description |
---|---|
name | string Name of DataSphere VM configuration. |
ExtendedWorkingStorage
Field | Description |
---|---|
type | enum StorageType |
size_gb | int64 |
File
Field | Description |
---|---|
desc | FileDesc |
sha256 | string SHA256 of the file. |
size_bytes | int64 File size in bytes. |
ExecuteProjectJobResponse
Field | Description |
---|---|
output_files[] | StorageFile Uploaded output files with URLs. |
result | JobResult Result of the job. |
StorageFile
Field | Description |
---|---|
file | File |
url | string File URL. |
JobResult
Field | Description |
---|---|
return_code | int64 Execution return code. |
Cancel
Cancels running job.
rpc Cancel (CancelProjectJobRequest) returns (google.protobuf.Empty
CancelProjectJobRequest
Field | Description |
---|---|
job_id | string ID of the job. |
reason | string Optional cancellation reason. |
ReadStdLogs
Returns stream of job logs.
rpc ReadStdLogs (ReadProjectJobStdLogsRequest) returns (stream ReadProjectJobStdLogsResponse)
ReadProjectJobStdLogsRequest
Field | Description |
---|---|
job_id | string ID of the job. |
offset | int64 Log offset. |
ReadProjectJobStdLogsResponse
Field | Description |
---|---|
logs[] | StdLog |
offset | int64 Log offset. |
StdLog
Field | Description |
---|---|
content | bytes Log contents. |
type | enum Type Log type.
|
ReadLogs
Returns stream of job logs.
rpc ReadLogs (ReadProjectJobLogsRequest) returns (stream ReadProjectJobLogsResponse)
ReadProjectJobLogsRequest
Field | Description |
---|---|
job_id | string ID of the job. |
offset | int64 Log offset. |
ReadProjectJobLogsResponse
Field | Description |
---|---|
logs[] | LogMessage |
offset | int64 Log offset. |
LogMessage
Field | Description |
---|---|
content | bytes Log message contents. |
created_at | google.protobuf.Timestamp Log message creation timestamp. |
source | oneof: standard_stream or file_path |
standard_stream | enum StandardStream Program standard streams.
|
file_path | string System debug log files. |
DownloadJobFiles
Returns download urls for job files.
rpc DownloadJobFiles (DownloadProjectJobFilesRequest) returns (DownloadProjectJobFilesResponse)
DownloadProjectJobFilesRequest
Field | Description |
---|---|
job_id | string Required. |
files[] | File The minimum number of elements is 1. |
File
Field | Description |
---|---|
desc | FileDesc |
sha256 | string SHA256 of the file. |
size_bytes | int64 File size in bytes. |
FileDesc
Field | Description |
---|---|
path | string Path of the file on filesystem. |
var | string Variable to use in cmd substitution. |
DownloadProjectJobFilesResponse
Field | Description |
---|---|
download_files[] | StorageFile |
StorageFile
Field | Description |
---|---|
file | File |
url | string File URL. |
List
Lists jobs.
rpc List (ListProjectJobRequest) returns (ListProjectJobResponse)
ListProjectJobRequest
Field | Description |
---|---|
project_id | string ID of the project. |
page_size | int64 The maximum number of results per page to return. If the number of available results is larger than page_size , the service returns a ListProjectJobResponse.page_token that can be used to get the next page of results in subsequent list requests. |
page_token | string Page token. To get the next page of results, set page_token to the ListProjectJobResponse.page_token returned by a previous list request. |
ListProjectJobResponse
Field | Description |
---|---|
jobs[] | Job Instances of the jobs. |
page_token | string This token allows you to get the next page of results for list requests. If the number of results is larger than ListProjectJobRequest.page_size, use the [next_page_token] as the value for the ListProjectJobRequest.page_token query parameter in the next list request. Each subsequent list request will have its own page_token to continue paging through the results. |
Job
Field | Description |
---|---|
id | string ID of the job. |
name | string Name of the job. |
desc | string Description of the job. |
created_at | google.protobuf.Timestamp Create job timestamp. |
started_at | google.protobuf.Timestamp Start job timestamp. |
finished_at | google.protobuf.Timestamp Finish job timestamp. |
status | enum JobStatus Status of the job. |
config | string Config of the job, copied from configuration file. |
created_by_id | string ID of the user who created the job. |
project_id | string ID of the project. |
job_parameters | JobParameters |
data_expires_at | google.protobuf.Timestamp Job data expiration timestamp. |
data_cleared | bool Marks if the job data has been cleared. |
output_files[] | File Output files of the job. |
log_files[] | File Job log files. |
diagnostic_files[] | File Job diagnostics files. |
data_size_bytes | int64 Job total data size. |
JobParameters
Field | Description |
---|---|
input_files[] | File List of input files. |
output_files[] | FileDesc List of output files descriptions. |
s3_mount_ids[] | string List of DataSphere S3 mount ids. |
dataset_ids[] | string List of DataSphere dataset ids. |
cmd | string Job run command. |
env | Environment Job environment description. |
attach_project_disk | bool Should project disk be attached to VM. |
cloud_instance_type | CloudInstanceType VM specification. |
extended_working_storage | ExtendedWorkingStorage Extended working storage configuration. |
FileDesc
Field | Description |
---|---|
path | string Path of the file on filesystem. |
var | string Variable to use in cmd substitution. |
Environment
Field | Description |
---|---|
vars | map<string,string> Environment variables. |
docker_image | oneof: docker_image_resource_id or docker_image_spec |
docker_image_resource_id | string DS docker image id. |
docker_image_spec | DockerImageSpec |
python_env | PythonEnv |
DockerImageSpec
Field | Description |
---|---|
image_url | string Docker image URL. |
username | string Username for container registry. |
password | oneof: password_plain_text or password_ds_secret_name Password for container registry. |
password_plain_text | string Plaintext password. |
password_ds_secret_name | string ID of DataSphere secret containing password. |
PythonEnv
Field | Description |
---|---|
conda_yaml | string Conda YAML. |
local_modules[] | File List of local modules descriptions. |
CloudInstanceType
Field | Description |
---|---|
name | string Name of DataSphere VM configuration. |
ExtendedWorkingStorage
Field | Description |
---|---|
type | enum StorageType |
size_gb | int64 |
File
Field | Description |
---|---|
desc | FileDesc |
sha256 | string SHA256 of the file. |
size_bytes | int64 File size in bytes. |
Get
Returns job by id.
rpc Get (GetProjectJobRequest) returns (Job)
GetProjectJobRequest
Field | Description |
---|---|
job_id | string ID of the job. |
Job
Field | Description |
---|---|
id | string ID of the job. |
name | string Name of the job. |
desc | string Description of the job. |
created_at | google.protobuf.Timestamp Create job timestamp. |
started_at | google.protobuf.Timestamp Start job timestamp. |
finished_at | google.protobuf.Timestamp Finish job timestamp. |
status | enum JobStatus Status of the job. |
config | string Config of the job, copied from configuration file. |
created_by_id | string ID of the user who created the job. |
project_id | string ID of the project. |
job_parameters | JobParameters |
data_expires_at | google.protobuf.Timestamp Job data expiration timestamp. |
data_cleared | bool Marks if the job data has been cleared. |
output_files[] | File Output files of the job. |
log_files[] | File Job log files. |
diagnostic_files[] | File Job diagnostics files. |
data_size_bytes | int64 Job total data size. |
JobParameters
Field | Description |
---|---|
input_files[] | File List of input files. |
output_files[] | FileDesc List of output files descriptions. |
s3_mount_ids[] | string List of DataSphere S3 mount ids. |
dataset_ids[] | string List of DataSphere dataset ids. |
cmd | string Job run command. |
env | Environment Job environment description. |
attach_project_disk | bool Should project disk be attached to VM. |
cloud_instance_type | CloudInstanceType VM specification. |
extended_working_storage | ExtendedWorkingStorage Extended working storage configuration. |
FileDesc
Field | Description |
---|---|
path | string Path of the file on filesystem. |
var | string Variable to use in cmd substitution. |
Environment
Field | Description |
---|---|
vars | map<string,string> Environment variables. |
docker_image | oneof: docker_image_resource_id or docker_image_spec |
docker_image_resource_id | string DS docker image id. |
docker_image_spec | DockerImageSpec |
python_env | PythonEnv |
DockerImageSpec
Field | Description |
---|---|
image_url | string Docker image URL. |
username | string Username for container registry. |
password | oneof: password_plain_text or password_ds_secret_name Password for container registry. |
password_plain_text | string Plaintext password. |
password_ds_secret_name | string ID of DataSphere secret containing password. |
PythonEnv
Field | Description |
---|---|
conda_yaml | string Conda YAML. |
local_modules[] | File List of local modules descriptions. |
CloudInstanceType
Field | Description |
---|---|
name | string Name of DataSphere VM configuration. |
ExtendedWorkingStorage
Field | Description |
---|---|
type | enum StorageType |
size_gb | int64 |
File
Field | Description |
---|---|
desc | FileDesc |
sha256 | string SHA256 of the file. |
size_bytes | int64 File size in bytes. |
Delete
Deletes specified job.
rpc Delete (DeleteProjectJobRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:DeleteProjectJobMetadata
Operation.response:google.protobuf.Empty
DeleteProjectJobRequest
Field | Description |
---|---|
job_id | string ID of the job. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
DeleteProjectJobMetadata
Field | Description |
---|---|
job_id | string ID of the job. |
DeleteData
Delete job data.
rpc DeleteData (DeleteProjectJobDataRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:DeleteProjectJobDataMetadata
Operation.response:DeleteProjectJobDataResponse
DeleteProjectJobDataRequest
Field | Description |
---|---|
job_id | string ID of the job. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
DeleteProjectJobDataMetadata
Field | Description |
---|---|
job_id | string ID of the job. |
DeleteProjectJobDataResponse
Empty.
DeleteAllData
Delete all jobs data.
rpc DeleteAllData (DeleteAllDataRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:DeleteAllDataMetadata
Operation.response:DeleteAllDataResponse
DeleteAllDataRequest
Field | Description |
---|---|
project_id | string |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
DeleteAllDataMetadata
Field | Description |
---|---|
project_id | string |
DeleteAllDataResponse
Empty.