Yandex Cloud
  • Services
  • Solutions
  • Why Yandex Cloud
  • Blog
  • Pricing
  • Documentation
  • Contact us
Get started
Language / Region
Yandex project
© 2023 Intertech Services AG
Yandex Data Transfer
  • Available transfers
  • Getting started
  • Step-by-step guide
  • Tutorials
  • Concepts
    • Relationships between service resources
    • Transfer types and lifecycles
    • What objects can be transferred
    • Yandex Data Transfer specifics for sources and targets
    • Operations on transfers
    • Network in Yandex Data Transfer
    • Speed for copying data in Yandex Data Transfer
    • Change data capture
    • Sharded copy
    • What tasks is the service used for?
    • Quotas and limits
  • Troubleshooting
  • Access management
  • Pricing policy
  • API reference
  • Questions and answers
  1. Concepts
  2. Relationships between service resources

Relationships between resources in Data Transfer

Written by
Yandex Cloud
  • Endpoint
  • Transfer
    • Transfer types
    • Compatibility of sources and targets

Yandex Data Transfer helps transfer data between DBMS, object storages, and message brokers. This way you can reduce the migration period and minimize downtime when switching to a new database.

Yandex Data Transfer is configurable via Yandex Cloud standard interfaces.

The service is suitable for creating a permanent replica of the database. The transfer of the database schema from the source to the target is automated.

Endpoint

Endpoint is a configuration used to connect to the datasource service or target service. In addition to connection settings, the endpoint may contain information about which data will be involved in the transfer and how it should be processed during the transfer.

The following can be the data source or target:

Service Source Target
Apache Kafka® topic: Your own topic or a topic in Managed Service for Apache Kafka®
AWS CloudTrail message stream
Your own BigQuery database
ClickHouse database: Your own or as part of the Managed Service for ClickHouse service
Greenplum® database: Your own or as part of the Managed Service for Greenplum® service
MongoDB database: Your own or as part of the Managed Service for MongoDB service
MySQL database: Your own or as part of the Managed Service for MySQL service
Your own Oracle database
PostgreSQL database: Your own or as part of the Managed Service for PostgreSQL service
S3-compatible bucket
Data stream Yandex Data Streams
Managed Service for YDB database: A database in Managed Service for YDB
Bucket Yandex Object Storage

Transfer

Transfer is the process of transmitting data between the source and target service. It should be in the same folder as the endpoints used.

If subnets are specified for endpoints, they must be hosted in the same availability zone. Otherwise, activating the transfer with such endpoints will result in an error.

Transfer types

The following types of transfers are available:

  • Snapshot: Transfers a snapshot of the source to the target. Apart from a one-time snapshot transfer, there are copy types, such as Regular and Regular incremental.
  • Increment: Continuously receives changes from the source and applies them to the target. Initial data synchronization is not performed.
  • Snapshot and increment: Transfers the current state of the source to the target and keeps it up-to-date.

For more information about the differences between transfer types, see Transfer types and lifecycles.

Compatibility of sources and targets

Possible source and target combinations:

Source \ Target Apache Kafka® ClickHouse Elasticsearch Greenplum® MongoDB MySQL OpenSearch PostgreSQL YDB Object Storage Source / Target
Airbyte® - C1 - - C1 C1 - C1 C1 - Airbyte®
Apache Kafka® R1 R1 R2 R1 - - R2 - R1 R1 Apache Kafka®
ClickHouse - C - - - - - - - - ClickHouse
Greenplum® - C - C1 - - - C1 - - Greenplum®
MongoDB - - - - CR - - - - C1 MongoDB
MySQL CR CR - - - CR - - CR1 C1 MySQL
Oracle - CR1 - - - - - CR1 - - Oracle
PostgreSQL CR CR - C1 - - - CR CR1 C1 PostgreSQL
YDB - C1 - - - - - - - C1 YDB
Yandex Data Streams - R1 R2 R1 R1 - R2 R1 R1 R1 Yandex Data Streams

C: Copy
R: Replicate
CR: Copy and replicate

1 This feature is in the Preview stage.
2 This feature is in the Preview stage and is available upon request. Contact support or your account manager.

Airbyte® endpoints

You can use Airbyte® to configure the following source endpoints:

  • AWS CloudTrail
  • BigQuery
  • Microsoft SQL Server
  • S3

Was the article helpful?

Language / Region
Yandex project
© 2023 Intertech Services AG
In this article:
  • Endpoint
  • Transfer
  • Transfer types
  • Compatibility of sources and targets