Creating a trigger for Data Streams that invokes a Serverless Containers container
Create a trigger for Data Streams that invokes a Serverless Containers container when data is sent to a stream.
Getting started
To create a trigger, you need:
-
A container that the trigger will invoke. If you do not have a container:
-
(Optional) A dead letter queue where messages that could not be processed by a container will be redirected. If you do not have a queue, create one.
-
Service accounts with rights:
- To invoke a container.
- To read from the stream that activates the trigger when data is sent there.
- (optional) To write to the dead letter queue.
You can use the same service account or different ones. If you do not have a service account, create one.
-
The stream that activates the trigger when it receives data. If you do not have a stream, create one.
Creating a trigger
Note
The trigger is initiated within 5 minutes of being created.
-
In the management console
, select the folder where you want to create your trigger. -
Open Serverless Containers.
-
In the left-hand panel, select
-
Click Create trigger.
-
Under Basic settings:
- Enter a name and description for the trigger.
- In the Type field, select
Data Streams
. - In the Launched resource field, select
Container
.
-
Under Data Streams settings, select a data stream and a service account with rights to read data from and write data to it.
-
(Optional) Under Batch message settings, specify:
- Message batch size in bytes. The values may range from 1 B to 64 KB. The default value is 1 B.
- Maximum wait time. The values may range from 1 to 60 seconds. The default value is 1 second.
The trigger groups messages for a period of time not exceeding the specified timeout and sends them to a container. The total amount of data transmitted to a container may exceed the specified group size if the data is transmitted as a single message. Otherwise, the amount of data does not exceed the batch size.
-
Under Container settings, select a container and specify a service account to invoke it under.
-
(Optional) Under Repeat request settings:
- In the Interval field, specify the time interval to retry invoking the container if the current attempt fails. The values may range from 10 to 60 seconds. The default value is 10 seconds.
- In the Number of attempts field, specify the number of invocation retries before the trigger moves a message to the Dead Letter Queue. The values may range from 1 to 5. The default value is 1.
-
(Optional) Under Dead Letter Queue settings, select the dead letter queue and the service account with write privileges for this queue.
-
Click Create trigger.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To create a trigger that invokes a container, run this command:
yc serverless trigger create yds \
--name <trigger_name> \
--database <database_location> \
--stream <data_stream_name> \
--batch-size 1b \
--batch-cutoff 1s \
--stream-service-account-id <service_account_ID> \
--invoke-container-id <container_ID> \
--invoke-container-service-account-id <service_account_ID> \
--retry-attempts 1 \
--retry-interval 10s \
--dlq-queue-id <dead_letter_queue_ID> \
--dlq-service-account-id <service_account_ID>
Where:
-
--name
: Trigger name. -
--database
: Location of the YDB database that the Data Streams stream is linked to.To find out where the database is located, run the
yc ydb database list
command. The DB location is specified in theENDPOINT
column, in thedatabase
parameter, e.g.,/ru-central1/b1gia87mbah2********/etn7hehf6gh3********
. -
--stream
: Data stream name. -
--batch-size
: Message batch size. This is an optional parameter. The values may range from 1 B to 64 KB. The default value is 1 B. -
--batch-cutoff
: Maximum wait time. This is an optional parameter. The values may range from 1 to 60 seconds. The default value is 1 second. The trigger groups messages for a period not exceedingbatch-cutoff
and sends them to a container. The total amount of data transmitted to a container may exceedbatch-size
if the data is transmitted as a single message. Otherwise, the amount of data does not exceedbatch-size
. -
--stream-service-account-id
: ID of the service account with rights to read from the data stream and write to it.
--invoke-container-id
: Container ID.--invoke-container-service-account-id
: Service account with rights to invoke the container.--retry-attempts
: Number of invocation retries before the trigger moves a message to the dead letter queue. This is an optional parameter. The values may range from 1 to 5. The default value is 1.--retry-interval
: Time to retry invoking the container if the current attempt fails. This is an optional parameter. The values may range from 10 to 60 seconds. The default value is 10 seconds.--dlq-queue-id
: Dead letter queue ID. This is an optional parameter.--dlq-service-account-id
: Service account with the permission to write messages to the dead letter queue. Optional parameter.
Result:
id: a1s5msktijh2********
folder_id: b1gmit33hgh2********
created_at: "2022-10-24T14:07:04.693126923Z"
name: data-streams-trigger
rule:
data_stream:
database: /ru-central1/b1gia87mbah2********/etn7hehh2********
stream: streams-name
service_account_id: ajep8qm0kh2********
batch_settings:
size: "1"
cutoff: 1s
invoke_container:
container_id: bba5jb38o8h2********
service_account_id: aje03adgd2h2********
retry_settings:
retry_attempts: "1"
interval: 10s
dead_letter_queue:
queue-id: yrn:yc:ymq:ru-central1:b1gmit33ngh2********:dlq
service-account-id: aje3lebfemh2********
status: ACTIVE
To create a trigger for Data Streams, use the create REST API method for the Trigger resource or the TriggerService/Create gRPC API call.
Checking the result
Make sure the trigger is working properly. To do this, view container logs that show information about invocations.