Yandex Cloud
  • Services
  • Solutions
  • Why Yandex Cloud
  • Blog
  • Pricing
  • Documentation
  • Contact us
Get started
Language / Region
Yandex project
© 2023 Intertech Services AG
Yandex Data Transfer
  • Available transfers
  • Getting started
  • Step-by-step guide
  • Tutorials
  • Concepts
  • Troubleshooting
  • Access management
  • Pricing policy
  • API reference
  • Questions and answers

Getting started with Data Transfer

Written by
Yandex Cloud
  • Before you begin
  • Configure the source and the target
    • Compatibility of sources and targets
    • Preparing sources and targets
  • Creating an endpoint for the source
  • Creating an endpoint for the target
  • Create a transfer
  • Activate the transfer
  • What's next

Copy or replicate data from a source to a target by creating endpoints and configuring transfers between them.

Before you begin

  1. Go to the Yandex Cloud console and select the folder where you want to perform the operations. If that folder doesn't exist, create it:

    Management console
    CLI
    API
    1. In the management console, select a cloud and click Create icon Create folder.

    2. Enter the folder name. Naming requirements:

      • The length can be from 3 to 63 characters.
      • It may contain lowercase Latin letters, numbers, and hyphens.
      • The first character must be a letter. The last character can't be a hyphen.
    3. (Optional) Enter a description of the folder.

    4. Select the Create a default network option. A network is created with subnets in each availability zone. A default security group, where all network traffic is allowed inside, will also be created in this network.

    5. Click Create.

    If you don't have the Yandex Cloud command line interface yet, install and initialize it.

    1. See a description of the create folder command:

      yc resource-manager folder create --help
      
    2. Create a new folder:

      • with a name and without a description:

        yc resource-manager folder create \
          --name new-folder
        
        • The length can be from 3 to 63 characters.
        • It may contain lowercase Latin letters, numbers, and hyphens.
        • The first character must be a letter. The last character can't be a hyphen.
      • with a name and description:

        yc resource-manager folder create \
          --name new-folder \
          --description "my first folder with description"
        

    Use the create method for the Folder resource of the Yandex Resource Manager service.

  2. On the billing page, make sure you linked a billing account and it has the ACTIVE or TRIAL_ACTIVE status. If you don't have a billing account, create one.

  3. On the Access management page, make sure you have the editor role or higher for the desired folder or the cloud that the folder belongs to.

Configure the source and the target

Different systems can act as a source and as a target.

Compatibility of sources and targets

Possible source and target combinations:

Source \ Target Apache Kafka® ClickHouse Elasticsearch Greenplum® MongoDB MySQL OpenSearch PostgreSQL YDB Object Storage Source / Target
Airbyte® - C1 - - C1 C1 - C1 C1 - Airbyte®
Apache Kafka® R1 R1 R2 R1 - - R2 - R1 R1 Apache Kafka®
ClickHouse - C - - - - - - - - ClickHouse
Greenplum® - C - C1 - - - C1 - - Greenplum®
MongoDB - - - - CR - - - - C1 MongoDB
MySQL CR CR - - - CR - - CR1 C1 MySQL
Oracle - CR1 - - - - - CR1 - - Oracle
PostgreSQL CR CR - C1 - - - CR CR1 C1 PostgreSQL
YDB - C1 - - - - - - - C1 YDB
Yandex Data Streams - R1 R2 R1 R1 - R2 R1 R1 R1 Yandex Data Streams

C: Copy
R: Replicate
CR: Copy and replicate

1 This feature is in the Preview stage.
2 This feature is in the Preview stage and is available upon request. Contact support or your account manager.

Airbyte® endpoints

You can use Airbyte® to configure the following source endpoints:

  • AWS CloudTrail
  • BigQuery
  • Microsoft SQL Server
  • S3

Preparing sources and targets

Prepare the source for sending data:

  • Airbyte®:
    • AWS CloudTrail
    • BigQuery
    • S3
  • Apache Kafka®
  • ClickHouse
  • Greenplum®
  • MongoDB
  • MySQL
  • PostgreSQL
  • Yandex Data Streams

Prepare the target for receiving the data:

  • ClickHouse

  • Greenplum®

  • MongoDB

  • MySQL

  • Object Storage

  • PostgreSQL

  • Yandex Managed Service for YDB

Creating an endpoint for the source

  1. Go to the folder page and select Yandex Data Transfer.
  2. Click Create endpoint.
  3. Make sure that the Direction field is set to Source.
  4. Enter a name for the endpoint.
  5. In the Database type field, select the type of the DBMS to transfer data from.
  6. Specify endpoint parameters in the corresponding settings section.
  7. Click Create.

For more information, see Endpoint management.

Creating an endpoint for the target

  1. Go to the folder page and select Yandex Data Transfer.
  2. Click Create endpoint.
  3. Make sure that the Direction field is set to Target.
  4. Enter a name for the endpoint.
  5. In the Database type field, select the type of DBMS that you want to transfer data to.
  6. Specify endpoint parameters in the corresponding settings section.
  7. Click Create.

For more information, see Endpoint management.

Create a transfer

  1. Go to the folder page and select Yandex Data Transfer.
  2. On the left-hand panel, select Transfers.
  3. Click Create transfer.
  4. Select the endpoint for the source and the endpoint for the target.
  5. Enter a name for the transfer.
  6. Select the type of transfer:
    • Snapshot: To create a full copy of the data without receiving further updates from the source. You can also use this type to replicate constantly changing tables.
    • Regular snapshot: To create a full copy of data at certain intervals of time.
    • Increment: To continuously receive data updates from the source and apply them to the target (without creating a full copy of the source data).
    • Snapshot and increment: To create a full copy of the source data and keep it up-to-date.
  7. (optional) Add a transfer description.
  8. Click Create.

For more information, see Types of transfers.

Activate the transfer

  1. Go to the folder page and select Yandex Data Transfer.
  2. On the left-hand panel, select Transfers.
  3. Click next to the name of the desired transfer and select Activate.

The data transfer process will begin.

For more information, see Managing the transfer process.

What's next

  • Read about service concepts.
  • Learn more about preparing databases for the service and setting up transfers.
  • Read questions and answers.

Greenplum® and Greenplum Database® are registered trademarks or trademarks of VMware, Inc in the United States and/or other countries.

Was the article helpful?

Language / Region
Yandex project
© 2023 Intertech Services AG
In this article:
  • Before you begin
  • Configure the source and the target
  • Compatibility of sources and targets
  • Preparing sources and targets
  • Creating an endpoint for the source
  • Creating an endpoint for the target
  • Create a transfer
  • Activate the transfer
  • What's next