Get started
Yandex Serverless Containers

Creating a trigger for Data Streams that invokes a Serverless Containers container

Written by

Create a trigger for Data Streams that invokes a Serverless Containers container when data is sent to a stream.

Before you begin

To create a trigger, you need:

  • A container that the trigger will invoke. If you don't have a container:

  • (optional) A Dead Letter Queue where messages that could not be processed by a container will be redirected. If you don't have a queue, create one.

  • Service accounts with rights:

    • To invoke a container.
    • To read from the stream that activates the trigger when data is sent there.
    • (optional) To write to the Dead Letter Queue.

    You can use the same service account or different ones. If you don't have a service account, create one.

  • The stream that activates the trigger when it receives data. If you don't have a stream, create one.

Creating a trigger

Note

The trigger is initiated within 5 minutes of being created.

  1. In the management console, select the folder where you wish to create your trigger.

  2. Open Serverless Containers.

  3. On the left-hand panel, select Triggers.

  4. Click Create trigger.

  5. Under Basic parameters:

    • Enter a name and description for the trigger.
    • In the Type field, select Data Streams.
    • In the Launched resource field, select Container.

  6. Under Data Streams settings, select a data stream and a service account with rights to read data from this data stream and write data to it.

  7. (optional) Under Batch message settings, specify:

    • Size of the message group in bytes. Valid values range from 1 B to 64 KB. The default is 1 B.
    • Maximum wait time. Valid values range from 1 to 60 seconds. The default is 1 second.

    The trigger groups messages for a period of time not exceeding the specified timeout and sends them to a container. The total amount of data passed to a container may exceed the specified group size if the data is transmitted as a single message. Otherwise, the amount of data does not exceed the group size.

  8. Under Container settings, select a container and specify:

  9. (optional) Under Repeat request settings:

    • In the Interval field, specify the time delay for the container to be invoked again if the current attempt fails. Values can be from 10 to 60 seconds. The default is 10 seconds.
    • In the Number of attempts field, specify the number of invocation retries before the trigger moves a message to the Dead Letter Queue. Values can be from 1 to 5. The default is 1.

  10. (optional) Under Dead Letter Queue settings, select the Dead Letter Queue and the service account with write privileges for this queue.

  11. Click Create trigger.

If you don't have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a trigger that invokes a container, run the command:

yc serverless trigger create yds \
  --name <trigger_name> \
  --database <database_location> \
  --stream <data_stream_name> \
  --batch-size 1b \
  --batch-cutoff 1s \
  --stream-service-account-id <service_account_ID> \
  --invoke-container-id <container_ID> \
  --invoke-container-service-account-id <service_account_ID> \
  --retry-attempts 1 \
  --retry-interval 10s \
  --dlq-queue-id <Dead_Letter_Queue_ID> \
  --dlq-service-account-id <service_account_ID>

Where:

  • --name: Trigger name.

  • --database: Location of the YDB database that the Data Streams stream is linked to.

    To find out where the database is located, run the yc ydb database list command. The DB location is specified in the ENDPOINT column, the database parameter, such as /ru-central1/b1gia87mba**********/etn7hehf6g*******.

  • --stream: Data stream name.

  • --batch-size: Message batch size. Optional. Valid values range from 1 B to 64 KB. The default is 1 B.

  • --batch-cutoff: Maximum waiting time. Optional. Valid values range from 1 to 60 seconds. The default is 1 second. The trigger groups messages for a period not exceeding batch-cutoff and sends them to a container. The total amount of data passed to a container may exceed batch-size if the data is transmitted as a single message. Otherwise, the amount of data does not exceed batch-size.

  • --stream-service-account-id: ID of the service account with rights to read from the data stream and write to it.

  • --invoke-container-id: Container ID.
  • --invoke-container-service-account-id: Service account with rights to invoke the container.
  • --retry-attempts: Time to retry invoking the container if the current attempt fails. Optional. Values can be from 10 to 60 seconds. The default is 10 seconds.
  • --retry-interval: The number of invocation retries before the trigger moves a message to the Dead Letter Queue. Optional. Values can be from 1 to 5. The default is 1.
  • --dlq-queue-id: Dead Letter Queue ID. Optional.
  • --dlq-service-account-id: Service account with rights to write messages to the Dead Letter Queue. Optional.

Result:

id: a1s5msktij**********
folder_id: b1gmit33hg**********
created_at: "2022-10-24T14:07:04.693126923Z"
name: data-streams-trigger
rule:
  data_stream:
    database: /ru-central1/b1gia87mba**********/etn7heh**********
    stream: streams-name
    service_account_id: ajep8qm0k**********
    batch_settings:
      size: "1"
      cutoff: 1s
    invoke_container:
      container_id: bba5jb38o8**********
      service_account_id: aje03adgd2**********
      retry_settings:
        retry_attempts: "1"
        interval: 10s
      dead_letter_queue:
        queue-id: yrn:yc:ymq:ru-central1:b1gmit33ng**********:dlq
        service-account-id: aje3lebfem**********
status: ACTIVE

You can create a trigger for Data Streams using the create.

Checking the result

Make sure the trigger is working properly. To do this, view container logs that show information about invocations.

See also

In this article: