Get started
Yandex Cloud Functions

Creating a trigger for Data Streams that invokes a Cloud Functions function

Written by

Create a trigger for Data Streams that calls a Cloud Functions function when data is sent to a stream.

Before you begin

To create a trigger, you need:

  • A function that the trigger will invoke. If you don't have a function:

  • (optional) A Dead Letter Queue where messages that could not be processed by a function will be redirected. If you don't have a queue, create one.

  • Service accounts with rights:

    • To invoke a function.
    • To read from the stream that activates the trigger when data is sent there.
    • (optional) To write to the Dead Letter Queue.

    You can use the same service account or different ones. If you don't have a service account, create one.

  • The stream that activates the trigger when it receives data. If you don't have a stream, create one.

Creating a trigger

Note

The trigger is initiated within 5 minutes of being created.

  1. In the management console, select the folder where you wish to create your trigger.

  2. Select Cloud Functions.

  3. On the left-hand panel, select Triggers.

  4. Click Create trigger.

  5. Under Basic parameters:

    • Enter a name and description for the trigger.
    • In the Type field, select Data Streams.
    • In the Launched resource field, select Function.

  6. Under Data Streams settings, select a data stream and a service account with rights to read data from this data stream and write data to it.

  7. (optional) Under Batch message settings, specify:

    • Size of the message group in bytes. Valid values range from 1 B to 64 KB. The default is 1 B.
    • Maximum wait time. Valid values range from 1 to 60 seconds. The default is 1 second.

    The trigger groups messages for a period of time not exceeding the specified timeout and sends them to a function. The total amount of data passed to a function may exceed the specified group size if the data is transmitted as a single message. Otherwise, the amount of data does not exceed the group size.

  8. Under Function settings, select a function and specify:

  9. (optional) Under Repeat request settings:

    • In the Interval field, specify the time after which the function will be invoked again if the current attempt fails. Values can be from 10 to 60 seconds. The default is 10 seconds.
    • In the Number of attempts field, specify the number of invocation retries before the trigger moves a message to the Dead Letter Queue. Values can be from 1 to 5. The default is 1.

  10. (optional) Under Dead Letter Queue settings, select the Dead Letter Queue and the service account with write privileges for this queue.

  11. Click Create trigger.

If you don't have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a trigger that invokes a function, run the command:

yc serverless trigger create yds \
  --name <trigger name> \
  --database <database location> \
  --stream <data stream name> \
  --batch-size 1b \
  --batch-cutoff 1s \
  --stream-service-account-id <service account ID> \
  --invoke-function-id <function ID> \
  --invoke-function-service-account-id <service account ID> \
  --retry-attempts 1 \
  --retry-interval 10s \
  --dlq-queue-id <Dead Letter Queue ID> \
  --dlq-service-account-id <service account ID>

Where:

  • --name: Trigger name.

  • --database: Location of the YDB database that the Data Streams stream is linked to.

    To find out where the database is located, run the yc ydb database list command. The DB location is specified in the ENDPOINT column, the database parameter, such as /ru-central1/b1gia87mba**********/etn7hehf6g*******.

  • --stream: Data Streams data stream name.

  • --batch-size: Message batch size. Optional. Valid values range from 1 B to 64 KB. The default is 1 B.

  • --batch-cutoff: Maximum waiting time. Optional. Valid values range from 1 to 60 seconds. The default is 1 second. The trigger groups messages for a period not exceeding batch-cutoff and sends them to a function. The total amount of data passed to a function may exceed batch-size if the data is transmitted as a single message. Otherwise, the amount of data does not exceed batch-size.

  • --stream-service-account-id: ID of the service account with rights to read from the data stream and write to it.

  • --invoke-function-id: Function ID.
  • --invoke-function-service-account-id: Service account with rights to invoke the function.
  • --retry-attempts: The time after which the function will be invoked again if the current attempt fails. Optional. Values can be from 10 to 60 seconds. The default is 10 seconds.
  • --retry-interval: The number of invocation retries before the trigger moves a message to the Dead Letter Queue. Optional. Values can be from 1 to 5. The default is 1.
  • --dlq-queue-id: Dead Letter Queue ID. Optional.
  • --dlq-service-account-id: Service account with rights to write messages to the Dead Letter Queue. Optional.

Result:

id: a1smnfisr5**********
folder_id: b1gc1t4cb6**********
created_at: "2022-10-31T10:57:08.234586266Z"
name: data-streams-trigger
rule:
  data_stream:
    database: /ru-central1/b1gvlrnlei**********/etn3ege6nj**********
    stream: yds-stream
    service_account_id: aje07l4q4v**********
    batch_settings:
      size: "1"
      cutoff: 1s
    invoke_function:
      function_id: d4e155orh3**********
      function_tag: $latest
      service_account_id: aje07l4q4v**********
      retry_settings:
        retry_attempts: "1"
        interval: 10s
      dead_letter_queue:
        queue_id: yrn:yc:ymq:ru-central1:b1gc1t4cb6**********:queue_dead
        service_account_id: aje07l4q4v**********
status: ACTIVE

You can create a trigger for Data Streams using the create.

Checking the result

Make sure the trigger is working properly. To do this, view function logs: they show information about calls.

See also

In this article: