Relationships between resources in Data Transfer
Yandex Data Transfer helps transfer data between DBMS, object storages, and message brokers. This way you can reduce the migration period and minimize downtime when switching to a new database.
Yandex Data Transfer is configurable via Yandex Cloud standard interfaces.
The service is suitable for creating a permanent replica of the database. The transfer of the database schema from the source to the target is automated.
Endpoint
Endpoint is a configuration used to connect to the datasource service or target service. In addition to connection settings, the endpoint may contain information about which data will be involved in the transfer and how it should be processed during the transfer.
The following can be the data source or target:
| Service | Source | Target |
|---|---|---|
| Apache Kafka® topic: Your own topic or a topic in Managed Service for Apache Kafka® | ||
| AWS CloudTrail message stream | ||
| Your own BigQuery database | ||
| ClickHouse database: Your own or as part of the Managed Service for ClickHouse service | ||
| Greenplum® database: Your own or as part of the Managed Service for Greenplum® service | ||
| MongoDB database: Your own or as part of the Managed Service for MongoDB service | ||
| MySQL database: Your own or as part of the Managed Service for MySQL service | ||
| Your own Oracle database | ||
| PostgreSQL database: Your own or as part of the Managed Service for PostgreSQL service | ||
| S3-compatible bucket | ||
| Data stream Yandex Data Streams | ||
| Managed Service for YDB database: A database in Managed Service for YDB | ||
| Bucket Yandex Object Storage |
Transfer
Transfer is the process of transmitting data between the source and target service. It should be in the same folder as the endpoints used.
If subnets are specified for endpoints, they must be hosted in the same availability zone. Otherwise, activating the transfer with such endpoints will result in an error.
Transfer types
The following types of transfers are available:
- Snapshot: Transfers a snapshot of the source to the target. Apart from a one-time snapshot transfer, there are copy types, such as Regular and Regular incremental.
- Increment: Continuously receives changes from the source and applies them to the target. Initial data synchronization is not performed.
- Snapshot and increment: Transfers the current state of the source to the target and keeps it up-to-date.
For more information about the differences between transfer types, see Transfer types and lifecycles.
Compatibility of sources and targets
Possible source and target combinations:
| Source \ Target | Apache Kafka® | ClickHouse | Elasticsearch | Greenplum® | MongoDB | MySQL | OpenSearch | PostgreSQL | YDB | Object Storage | Source / Target |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Airbyte® | - | C1 | - | - | C1 | C1 | - | C1 | C1 | - | Airbyte® |
| Apache Kafka® | R1 | R1 | R2 | R1 | - | - | R2 | - | R1 | R1 | Apache Kafka® |
| ClickHouse | - | C | - | - | - | - | - | - | - | - | ClickHouse |
| Greenplum® | - | C | - | C1 | - | - | - | C1 | - | - | Greenplum® |
| MongoDB | - | - | - | - | CR | - | - | - | - | C1 | MongoDB |
| MySQL | CR | CR | - | - | - | CR | - | - | CR1 | C1 | MySQL |
| Oracle | - | CR1 | - | - | - | - | - | CR1 | - | - | Oracle |
| PostgreSQL | CR | CR | - | C1 | - | - | - | CR | CR1 | C1 | PostgreSQL |
| YDB | - | C1 | - | - | - | - | - | - | - | C1 | YDB |
| Yandex Data Streams | - | R1 | R2 | R1 | R1 | - | R2 | R1 | R1 | R1 | Yandex Data Streams |
C: Copy
R: Replicate
CR: Copy and replicate
1 This feature is in the Preview stage.
2 This feature is in the Preview stage and is available upon request. Contact support or your account manager.
Airbyte® endpoints
You can use Airbyte® to configure the following source endpoints: