Background operations
Warning
The Serverlesss mode will be discontinued on March 1, 2024.
In Serverlesss mode in DataSphere, you can run such time-consuming operations as model training, in the background. To do this, use special cells where your code is executed asynchronously. In this case, you can continue working with a notebook.
If another part of a notebook uses the same variable as an asynchronous operation, a notification appears in the notebook, and you will need to specify the variable value explicitly when the asynchronous operation is complete.
Note
Background operations are not available in Dedicated mode.
Specifics of background operations:
-
Running operations in the background does not guarantee their immediate execution.
-
In general, background operations may take longer than regular ones.
-
Background operations can run on preemptible virtual machines and resources.
-
Any background operations are suspended if there is an attempt to run interactive functions (such as,
input()
orgetpass()
). -
Background operations are priced differently. For more information, see DataSphere pricing.
-
Cells with running background operations have read-only access to project storage. If there is a conflict in variable values after a background operation is complete, DataSphere will ask whether to save the computation result or to roll back to the previous values.
Tip
Use object storage to save data, such as logs or intermediate results, during a background operation. To learn how to use S3 storage in DataSphere projects, see Connecting to S3 using the boto3 library.
Running background operations
To run a background operation, add the #pragma async
comment to the cell.
To run a test background operation:
-
Specify a test model, such as:
import tensorflow as tf import datetime mnist = tf.keras.datasets.mnist (x_train, y_train),(x_test, y_test) = mnist.load_data() x_train, x_test = x_train / 255.0, x_test / 255.0 def create_model(): return tf.keras.models.Sequential([ tf.keras.layers.Flatten(input_shape=(28, 28)), tf.keras.layers.Dense(512, activation='relu'), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(10, activation='softmax') ])
-
Create and compile a model:
model = create_model() model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
-
Start model training by adding the
#pragma async
comment at the beginning of the cell:#pragma async model = create_model() model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S") tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1) model.fit(x=x_train, y=y_train, epochs=5, validation_data=(x_test, y_test), callbacks=[tensorboard_callback])
While the model is being trained, you can perform operations in other cells.
Aborting background operations
To abort an operation:
- Select the cell where you want to abort an operation.
- In the top panel of the project window, click Kernel.
- In the menu that opens, click Interrupt selected cell(s).