Yandex Cloud
  • Services
  • Solutions
  • Why Yandex Cloud
  • Blog
  • Pricing
  • Documentation
  • Contact us
Get started
Language / Region
Yandex project
© 2023 Intertech Services AG
Yandex Data Transfer
  • Available transfers
  • Getting started
  • Step-by-step guides
    • All guides
    • Preparing for the transfer
    • Configuring endpoints
      • Endpoint management
      • Configuring source endpoints
      • Configuring target endpoints
    • Managing transfer process
    • Working with databases during transfer
    • Monitoring transfer status
  • Tutorials
  • Concepts
  • Troubleshooting
  • Access management
  • Pricing policy
  • API reference
  • FAQ
  1. Step-by-step guides
  2. Configuring endpoints
  3. Endpoint management

Endpoint management

Written by
Yandex Cloud
  • Getting a list of endpoints
  • Creating an endpoint
  • Updating the endpoint
  • Deleting an endpoint

The source endpoint describes the settings of the database that information will be transferred from using Data Transfer and the target endpoint the database you are going to transfer the data to. You can create, edit, or delete such endpoints.

Getting a list of endpoints

Management console
  1. Go to the folder page and select Yandex Data Transfer.
  2. In the left-hand panel, select Endpoints.

Creating an endpoint

Management console
CLI
Terraform
API
  1. Go to the folder page and select Yandex Data Transfer.
  2. Click Create endpoint.
  3. In the Direction field, select Source or Target.
  4. Enter a name for the endpoint. Use lowercase Latin letters and numbers.
  5. (optional) Enter a description of the endpoint.
  6. Select the appropriate value in the Database type field. An endpoint of the corresponding type will be created.
  7. Set the endpoint parameters:
    • Sources:
      • Airbyte®:
        • AWS CloudTrail
        • BigQuery
        • MS SQL Server
        • S3
      • Apache Kafka®
      • ClickHouse
      • EventHub
      • Greenplum®
      • Yandex Metrica
      • MongoDB
      • MySQL
      • Oracle
      • PostgreSQL
      • Yandex Data Streams
      • Yandex Managed Service for YDB
    • Targets:
      • Apache Kafka®
      • ClickHouse
      • Elasticsearch
      • Greenplum®
      • MongoDB
      • MySQL
      • Object Storage
      • OpenSearch
      • PostgreSQL
      • Yandex Data Streams
      • Yandex Managed Service for YDB
  8. Click Create endpoint.

Note

You can create endpoints using the CLI commands only for MySQL and PostgreSQL sources and targets.

If you don't have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create an endpoint:

  1. View a description of the CLI create endpoint command:

    yc datatransfer endpoint create --help
    
  2. Specify the endpoint parameters in the create command (only some of the supported parameters are given in the example):

    yc datatransfer endpoint create <endpoint type> \
       --name <endpoint name> \
       --description <endpoint description> \
       <endpoint parameters>
    

    You can view the endpoint type and parameters in the settings section for the appropriate data source or target.

You can create endpoints with Terraform only for MySQL, PostgreSQL, MongoDB, and ClickHouse sources and targets.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it by configuration files. They store the infrastructure description in HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Mozilla Public License.

For more information about the provider resources, see the documentation on the Terraform site or mirror site.

If you change the configuration files, Terraform automatically determines which part of your configuration is already deployed and what should be added or removed.

To create an endpoint:

  1. Using the command line, navigate to the folder that will contain the Terraform configuration files with an infrastructure plan. Create the directory if it does not exist.

    1. If you do not have Terraform yet, install it and create a configuration file with provider settings.
  2. Create a configuration file with a description of your endpoint.

    Example of the configuration file structure:

    resource "yandex_datatransfer_endpoint" "<endpoint name in Terraform>" {
      name = "<endpoint name>"
      settings {
        <endpoint type> {
          <endpoint parameters>
        }
      }
    }
    
  3. You can view the endpoint type and parameters in the settings section for the appropriate data source or target.

  4. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

  5. Confirm the resources have been updated.

    1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply
        
      2. Confirm the update of resources.

      3. Wait for the operation to complete.

  6. For more information, see the Terraform provider documentation.

Note

You can create endpoints with the API only for MySQL and PostgreSQL sources and targets.

Use the create API method and include the following information in the request:

  • ID of the folder to host the endpoint, in the folderId parameter.
  • Endpoint name in the name parameter.
  • Endpoint description in the description parameter.
  • Endpoint parameters in the settings parameter.

You can view the endpoint parameters in the settings section for the appropriate data source or target.

Updating the endpoint

Management console
Terraform
API
  1. Go to the folder page and select Yandex Data Transfer.
  2. Select an endpoint and click Edit in the top panel.
  3. Edit the endpoint parameters:
    • Sources:
      • Airbyte®:
        • AWS CloudTrail
        • BigQuery
        • MS SQL Server
        • S3
      • Apache Kafka®
      • ClickHouse
      • EventHub
      • Greenplum®
      • Yandex Metrica
      • MongoDB
      • MySQL
      • Oracle
      • PostgreSQL
      • Yandex Data Streams
      • Yandex Managed Service for YDB
    • Targets:
      • Apache Kafka®
      • ClickHouse
      • Greenplum®
      • MongoDB
      • MySQL
      • Object Storage
      • PostgreSQL
      • Yandex Data Streams
      • Yandex Managed Service for YDB
  4. Click Apply.
  1. Open the current Terraform configuration file with the endpoint description.

    To learn how to create such a file, see Create endpoint.

  2. Edit the value in the name field (endpoint name) and the endpoint parameters under settings.

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

  4. Confirm the resources have been updated.

    1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply
        
      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Note

You can edit endpoints using the API only for MySQL and PostgreSQL sources and targets.

Use the update API method and include the following in the request:

  • Endpoint ID in the endpointId parameters.
  • Endpoint name in the name parameter.
  • Endpoint description in the description parameter.
  • Endpoint parameters in the settings parameter.

To get the endpoint ID:

  1. Go to the folder page in the management console and select Yandex Data Transfer.
  2. Click the endpoint you need.

You can view the endpoint parameters in the settings section for the appropriate data source or target.

Editing the endpoint settings of a transfer with the Replication type and the Replicating status will result in restarting the transfer.

Deleting an endpoint

Warning

Before you delete an endpoint, delete all the transfers it's included in.

To delete an endpoint:

Management console
CLI
Terraform
API
  1. Go to the folder page and select Yandex Data Transfer.
  2. Select the endpoint to delete.
  3. Click Delete in the top panel.
  4. In the window that opens, click Delete.

If you don't have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To delete an endpoint, run the command:

yc datatransfer endpoint delete <endpoint ID>

To get the endpoint ID:

  1. Go to the folder page in the management console and select Yandex Data Transfer.
  2. Click the endpoint you need.

To delete an endpoint created using Terraform:

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating an endpoint.

  2. Make sure that the endpoint being deleted is not bound to a transfer.

  3. Delete the endpoint description.

  4. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

  5. Type the word yes, then press Enter.

    1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply
        
      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Use the delete API method and provide the ID of the required endpoint in the endpointID request parameter:

To get the endpoint ID:

  1. Go to the folder page in the management console and select Yandex Data Transfer.
  2. Click the endpoint you need.

Greenplum® and Greenplum Database® are registered trademarks or trademarks of VMware, Inc in the United States and/or other countries.

Was the article helpful?

Language / Region
Yandex project
© 2023 Intertech Services AG
In this article:
  • Getting a list of endpoints
  • Creating an endpoint
  • Updating the endpoint
  • Deleting an endpoint