Get started
Yandex Data Transfer

Configuring a Apache Kafka® target endpoint

Written by

When creating or editing an endpoint, you can define:

Managed Service for Apache Kafka® cluster

Connecting to the database with the cluster ID specified in Yandex Cloud. Available only for clusters deployed in Yandex Managed Service for Apache Kafka®.

  • Managed Kafka: Select the cluster to connect to.

    • Username: Specify the username that Data Transfer will use to connect to the database.

    • Password: Enter the user's password to the database.

Custom installation

Connecting to the database with explicitly specified network addresses.

  • Broker URI: Specify broker host IPs or FQDNs.

    If the Apache Kafka® port number differs from the standard one, specify it with a colon after the host name:

    <broker host IP or FQDN>:<port number>

  • SSL: Use encryption to protect the connection.

  • PEM certificate: If transmitted data needs to be encrypted, for example, to meet the requirements of PCI DSS, upload the certificate file or add its contents as text.

  • Authentication: Select the SASL or Without authentication type.

    If SASL is selected, specify the hashing mechanism and the password and username of the account that Data Transfer will connect to the topic under.

Topic settings Apache Kafka®

  • Apache Kafka topic settings:

    • Topic full name: Specify the name of the topic to send messages to.

    • Topic prefix: Specify the topic prefix, similar to the Debezium database.server.name setting. Messages will be sent to a topic named <topic_prefix>.<schema>.<table_name>.

Yandex Data Transfer supports CDC for transfers from PostgreSQL, MongoDB, and MySQL databases to Apache Kafka® and Yandex Data Streams. Data is sent to the target in Debezium format. For more information about CDC mode, see Change data capture.

  • Save tx order: Do not split an event stream into independent queues by table. This setting is only compatible with the Topic full name setting.
In this article: