Configuring a Yandex Data Streams source endpoint

Database: Select a Yandex Managed Service for YDB DB registered in Yandex Data Streams as a source.

Stream: Specify the name of the stream associated with the database.

Service account: Select or create a service account with the yds.editor role that Data Transfer will use to connect to the data source.

Transformation rules

The rules used by Cloud Function to process an incoming stream:

• Processing function: Select one of the functions created in Yandex Cloud Functions.

  • Service account: Select or create a service account that the processing function will start under.

• Number of attempts: Set the number of attempts to invoke the processing function.

• Buffer size to send: Set the size of the buffer (in bytes) which when full data will be transferred to the processing function.

  The maximum buffer size is 3.5 MB. For more information about restrictions that apply when working with functions in Cloud Functions, see the corresponding section.

• Sending interval: Set the duration of the interval (in seconds) after the expiration of which the data from the stream should be transferred to the processing function.

  Note

  If the buffer becomes full or the sending interval expires, the data is transferred to the processing function.

• Call timeout: Set the allowed timeout of the response from the processing function (in seconds).

Conversion rules:

• Data format: Select one of the available formats:

  • JSON
  • CSV

• Data schema: Specify the schema as a list of fields or upload a file with a description of the schema in JSON format.

  Sample data schema

  [
      {
          "name": "request",
          "type": "string"
        }
  ]
  

• Add unmarked columns: Select this option to have the fields missing in the schema appear in the _rest column.

• Allow null in key columns: Select this option to allow the null value in key columns.