Saving a stream from Yandex Data Streams to Yandex Object Storage using Yandex Data Transfer

With Data Transfer, you can move data from a Data Streams data stream to an Object Storage bucket.

1. Prepare a Data Streams data stream.
2. Prepare and activate the transfer.
3. Test the transfer.

If you no longer need these resources, delete them.

Before you begin

Prepare the infrastructure:

Prepare a Data Streams data stream

1. Create a Data Streams data stream.

2. Send test data to the data stream. Use data from the vehicle's sensors in JSON format as a message:

{
    "device_id":"iv9a94th6rztooxh5ur2",
    "datetime":"2020-06-05 17:27:00",
    "latitude":"55.70329032",
    "longitude":"37.65472196",
    "altitude":"427.5",
    "speed":"0",
    "battery_voltage":"23.5",
    "cabin_temperature":"17",
    "fuel_level":null
}

Prepare and activate the transfer

1. Create a source endpoint:

   • Database type: Yandex Data Streams.

   • Endpoint parameters:

     • Connection settings:

       • Database: Select a Managed Service for YDB database from the list.
       • Data stream: Specify the name of the Data Streams data stream.
       • Service account: Select or create a service account with the yds.editor role.

     • (optional) Advanced settings → Conversions rules:

       • Data format: JSON.

       • Data schema: JSON specification:

         Create and upload the json_schema.json file in JSON format:

         json_schema.json

         [
             {
                 "name": "device_id",
                 "type": "string"
             },
             {
                 "name": "datetime",
                 "type": "datetime"
             },
             {
                 "name": "latitude",
                 "type": "double"
             },
             {
                 "name": "longitude",
                 "type": "double"
             },
             {
                 "name": "altitude",
                 "type": "double"
             },
             {
                 "name": "speed",
                 "type": "double"
             },
             {
                 "name": "battery_voltage",
                 "type": "any"
             },
             {
                 "name": "cabin_temperature",
                 "type": "double"
             },
             {
                 "name": "fuel_level",
                 "type": "any"
             }
         ]
         

2. Create a target endpoint:

   • Database type: Object Storage.

   • Endpoint parameters:

     • Connection settings:

       • Bucket: Specify the name of the bucket Object Storage.
       • Service account: Select or create a service account with the storage.uploader role.

     • Output format: Select JSON or CSV if you have enabled Conversion rules in the source endpoint settings.

3. Create a transfer:

   Manually

   Using Terraform

   1. Create a transfer of the Increment type that will use the created endpoints.
   2. Activate it.

   1. In the data-transfer-yds-obj.tf file, uncomment:

      • Variables:

        • source_endpoint_id and set it to the value of the endpoint ID for the source.
        • target_endpoint_id and set it to the value of the endpoint ID for the target.

      • yandex_datatransfer_transfer resource.

   2. Make sure the Terraform configuration files are correct using the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

   3. Create the required infrastructure:

      1. Run the command to view planned changes:

         terraform plan
         

         If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

         1. Run the command:

            terraform apply
            

         2. Confirm the update of resources.

         3. Wait for the operation to complete.

      Once created, a transfer is activated automatically.

Test the transfer

1. Wait for the transfer status to change to Replicating.

2. Make sure that the data from the Data Streams stream has been moved to the bucket Object Storage:

   1. In the management console, select the folder where the bucket is located.
   2. In the list of services, select Object Storage.
   3. Select the bucket from the list.
   4. Check that the bucket includes the file <stream name>_0.raw (.json or .csv, depending on the selected output format) with the test data.

3. Send a new message to a Data Streams stream:

   {
       "device_id": "rhibbh3y08qmz3sdbrbu",
       "datetime": "2020-06-06 09:49:54",
       "latitude": 55.71294467,
       "longitude": 37.66542005,
       "altitude": 429.13,
       "speed": 55.5,
       "battery_voltage": null,
       "cabin_temperature": 18,
       "fuel_level": 32
   }
   

4. Make sure that the following data has been added to the Object Storage bucket:

   1. In the management console, select the folder where the bucket is located.
   2. In the list of services, select Object Storage.
   3. Select the bucket from the list.
   4. Make sure that the file <stream name>_0.raw (.json or .csv, depending on the selected output format) has been added to the bucket and includes the new data.

Delete the resources you created

If you no longer need these resources, delete them:

1. Delete the transfer.
2. Delete endpoints for the source and target.
3. Delete the objects from the Object Storage bucket:

Delete the other resources, depending on the method used to create them: