Status monitoring of geographically distributed devices

In this scenario, you'll configure status monitoring for devices (such as vending machines) that are connected to Yandex IoT Core and located at various sites in a city. Sensors are emulated using the Yandex Cloud Functions service. If your sensors are connected, use them. You can monitor the status of vending machines on the Yandex DataLens map and charts.

The source code of this scenario is available on GitHub.

To configure monitoring of sensor readings in the server room:

1. Before you start.
2. Required paid resources.
3. Create the required Yandex IoT Core resources.
   1. Create a registry.
   2. Create devices.
4. Create a device emulator in Yandex Cloud Functions.
   1. Create a function to emulate data transmission from devices.
   2. Test the data transmission emulation function.
   3. Create a trigger to call the emulation function once per minute.
5. Create a cluster in Yandex Managed Service for PostgreSQL.
6. Create a data processing function in Yandex Cloud Functions.
   1. Create a function for processing data received.
   2. Test the data processing function.
   3. View the data processing result in Managed Service for PostgreSQL.
   4. Create a trigger to call the data processing function.
   5. View the trigger result in Managed Service for PostgreSQL.
7. Configure monitoring in Yandex DataLens.
   1. Set up a connection to Managed Service for PostgreSQL.
   2. Create a dataset.
   3. Create a chart for temperature and mains power readings.
   4. Create a chart with a map.
   5. Create a dashboard.

If you no longer need the created resources, delete them.

Before you start

If you don't have the Yandex.Cloud command line interface yet, install and initialize it.

Make sure the billing account is active:

1. Go to the management console. Then log in to Yandex.Cloud or sign up if don't already have an account.
2. On the billing page, make sure you linked a billing account, and it has the ACTIVE or TRIAL_ACTIVE status. If you don't have a billing account, create one.

If you have an active billing account, you can create or select a folder on the Yandex.Cloud page.

Learn more about clouds and folders.

Required paid resources

The cost includes:

• Fee for the number of Yandex IoT Core messages (see pricing).
• Fee for the number of Yandex Cloud Functions function calls (see pricing.
• Fee for computing resources and cluster storage in Yandex Managed Service for PostgreSQL (see pricing).
• Fee for exchanging data with devices in Yandex DataLens (see pricing).

Create the necessary resources Yandex IoT Core

To use Yandex IoT Core, you need to create a registry and devices that will exchange data and commands.

Create a registry and configure authorization using your login and password

To create a registry:

1. In management console, select the folder where you are running the scenario.

2. Select Yandex IoT Core.

3. Click Create registry.

4. In the Name field, enter a name for the registry. For example, my-registry.

5. In the Password field, set the password to access the registry.

   The password must be at least 14 characters long and contain lowercase letters, capital letters, and numbers.
   To create a password, you can use a password generator.

6. Save the password locally or remember it. The service doesn't show the password after you create it.

7. (optional) In the Description field, add additional information about the registry.

8. Click Create.

You can also complete authorization using certificates. Learn more about authorization in Yandex IoT Core.

Create a device and configure authorization using your login and password

Create three devices: my-device-1, my-device-2, and my-device-3.

To create a device:

1. In management console, select the folder where you are running the scenario.

2. Select Yandex IoT Core.

3. Select the registry you created in the previous step.

4. In the left part of the window, select Devices.

5. Click Add device.

6. In the Name field, enter the device name. For example, my-device-1.

7. In the Password field, set the password to access the device.

   To create a password, you can use a password generator.
   The password must be at least 14 characters long and contain lowercase letters, capital letters, and numbers.

8. Save the password locally or remember it. The service doesn't show the password after you create it.

9. (optional) In the Description field, add additional information about the device.

10. (optional) Add aliases:

    1. Click Add alias.

    2. Click Edit.

    3. Fill in the fields: enter an alias (for example, events) and the topic type after $devices/<deviceID> (for example, events).

       You can use the events alias instead of the $devices/<deviceID>/events topic.

    4. Repeat the steps for each alias you add.

11. Click Create.

12. Repeat the steps for each device you want to create.

You can also complete authorization using certificates. Learn more about authorization in Yandex IoT Core.

Create a device emulator in Cloud Functions

The emulator will send data from devices to the Yandex Managed Service for PostgreSQL cluster.

You'll need:

• Create and test the function that emulates transmitting data from each device sensor.
• Create a trigger that invokes the emulation function once per minute.

Create a function that emulates transmitting data from the device

To create a function:

1. In management console, select the folder where you are running the scenario.
2. Select Cloud Functions.
3. In the left part of the window, select Functions.
4. Click Create function.
5. In the Name field, enter the name of the function. For example, my-device-emulator-function.
6. (optional) In the Description field, add additional information about the function.
7. Click Create.
8. In the Editor window that opens, select nodejs12 from the Runtime environment list.
9. Select Method: Code editor.
10. In the left part of the Code editor window, click Create file.
11. In the New file window that opens, enter the name device-emulator.js.
12. Click Create.
13. Select the created file in the left part of the Code editor window.
14. In the right part of the Code editor window, insert the function code from GitHub.
15. In the Editor window, enter device-emulator.handler in the Entry point field.
16. In the Timeout, sec field, enter 10.
17. In the Memory field, leave the value 128 MB.
18. Create a service account that the function will use to send data to Yandex IoT Core:
    1. Click Create an account.
    2. In the Create service account window that opens, enter an account name in the Name field. For example, my-emulator-function-service-account.
    3. Add the viewer and iot.devices.writer roles for retrieving the device list and writing to resources:
       1. Click the icon.
       2. Select a role from the list.
       3. Click Create.
19. Set Environment variables for each sensor in the server room:

    1. Click Add environment variable.

    2. Fill in the Key and Value fields for the environment variables:

       Key	Description	Value
       CASH_DRAWER_SENSOR_VALUE	Fullness percentage of the cash drawer.	67.89
       COIN_DRAWER_SENSOR_VALUE	Fullness percentage of the coin drawer.	67.89
       TEMPERATURE_SENSOR_VALUE	Baseline temperature in the pickup box.	10.34
       POWER_SENSOR_VALUE	Baseline mains voltage.	24.12
       SERVICE_DOOR_SENSOR_VALUE	State of the open service door sensor.	False
       ITEM1_SENSOR_VALUE	Stock of product 1.	50.65
       ITEM2_SENSOR_VALUE	Stock of product 2.	80.97
       ITEM3_SENSOR_VALUE	Stock of product 3.	30.33
       ITEM4_SENSOR_VALUE	Stock of product 4.	15.15
       REGISTRY_ID	ID of the registry you created.	See the service management console Yandex IoT Core

20. In the upper-right part of the window, click Create version.

Test the emulation function

To test the function:

1. (optional) To get detailed information from the sensors, subscribe the registry to the Yandex IoT Core device topic, where $devices/<deviceID>/events/ is a device topic, and <deviceID> is the device ID in the service:

   CLI

   Run the command:

   $ yc iot mqtt subscribe \
         --username <registry ID> \
         --password <registry password> \
         --topic '$devices/<device ID>/events' \
         --qos 1
   

   Where:

   • --username and --password: Parameters for authorization using a username and password.
   • --topic: Device topic for sending data.
   • --message: Message text.
   • --qos: Quality of service (QoS).

   Learn more about subscribing to device topics in Yandex IoT Core.

2. In management console, select the folder where you are running the scenario.

3. Select Cloud Functions.

4. In the left part of the window, select Testing.

5. In the Version tag field, select $latest (most recent function).

6. Click Run test.

If the function is successful, the Function status field shows Done and the Function output field shows the following:

{
"statusCode" : 200
}

If you subscribe to the Yandex IoT Core device topic, you get the following JSON:

{
"DeviceId":"arealt9f3jh445it1laq",
"TimeStamp":"2020-07-21T22:38:12Z",
"Values":[
    {"Type":"Bool","Name":"Service door sensor","Value":"false"},
    {"Type":"Float","Name":"Power Voltage","Value":"24.12"},
    {"Type":"Float","Name":"Temperature","Value":"10.34"},
    {"Type":"Float","Name":"Cash drawer fullness","Value":"67.89"},
    {"Type":"Float","Name":"Coin drawer fullness","Value":"67.89"},
    {"Items":[
       {"Type":"Float", "Id":"1","Name":"Item 1","Fullness":"50.65"},
       {"Type":"Float", "Id":"2","Name":"Item 2","Fullness":"80.97"},
       {"Type":"Float", "Id":"3","Name":"Item 3","Fullness":"30.33"},
       {"Type":"Float", "Id":"4","Name":"Item 4","Fullness":"15.15"},
	]}
	]
}

Learn more about MQTT topics in Yandex IoT Core.

Create a trigger that invokes the function once per minute

To create a trigger:

1. In management console, select the folder where you are running the scenario.
2. Select Cloud Functions.
3. Select Triggers.
4. Click Create trigger.
5. In the Name field, enter a name for the trigger. For example, my-emulator-function-trigger.
6. (optional) In the Description field, add additional information about the trigger.
7. Select Type: Timer.
8. In the Cron expression field, enter * * * * ? * (invoke once per minute).
9. Under Function settings, enter the parameters previously set for the function:
   • Function: my-device-emulator-function.
   • Function version tag: $latest.
   • Service account: my-emulator-function-service-account.
10. (optional) Set parameters under Repeat request settings and Dead Letter Queue settings. They provide data security.
    • Repeat request settings are useful if you want to recall the function when the current function request fails.
    • Dead Letter Queue settings are necessary to forward messages that consumers couldn't process in standard queues.
      You can configure a standard message queue as a DLQ. If you haven't created a message queue yet, create one in Yandex Message Queue.
11. Click Create trigger.

Create a cluster in Yandex Managed Service for PostgreSQL

The example is based on the minimum values of host parameters. For real tasks, it's better to choose hosts with a guaranteed 100% vCPU share.

To create a cluster:

1. In management console, select the folder where you are running the scenario.

2. Select Yandex Managed Service for PostgreSQL.

3. Select Clusters.

4. Click Create cluster.

5. Under Basic parameters, enter, for example, my-pg-database in the Cluster name field.

6. (optional) In the Description field, add additional information about the cluster.

7. In the Environment list, select PRODUCTION.

8. In the Version list, select 12.

9. Under Host class, select Intel Cascade Lake in the Platform list.

10. Select the VM type: on the burstable tab, select b2.nano.

11. Under Storage size, go to the network-hdd tab.

12. Set the storage size to 10 GB.

13. Under Database, enter:

    • In the DB name field, enter db1.

    • In the User name field, enter user1.

    • In the Password field, set the password to access the database.

      Don't forget to save your password, you'll need it.

14. Don't change the values in the Locale for sorting and Charset locale fields. The default value is C.

15. Under Network, select default in the list.

16. Under Hosts, set up a connection to your database via a public IP address (this is necessary to access the database from Yandex Cloud Functions).

    1. On the right side of the availability zone line, click .
    2. In the window that opens, select an Availability zone. For example, ru-central1-a.
    3. Select a Subnet. For example, default-ru-cetral1-a.
    4. Enable Public access.
    5. Click Save.

17. Enable DataLens access.

18. Click Create cluster.

It takes several minutes to create a cluster. As a result, a window with cluster details appears.

Create a function for processing data in Yandex Cloud Functions

The function processes data from devices.

Create a function for processing received data

To create a function:

1. In management console, select the folder where you are running the scenario.
2. Select Cloud Functions.
3. In the left part of the window, select Functions.
4. Click Create function.
5. In the Name field, enter the name of the function. For example, my-db-function.
6. (optional) In the Description field, add additional information about the function.
7. Click Create.
8. In the Editor window that opens, select python37 in the Runtime environment list.
9. Select Method: go to the Code editor tab.
10. In the left part of the Code editor window, click Create file.
11. In the New file window that opens, enter the name myfunction.py.
12. Click Create.
13. In the left part of the Code editor window, select the created file.
14. In the right part of the window, insert the function code from GitHub.
15. In the Entry point field of the Editor window, specify myfunction.msgHandler.
16. In the Timeout, sec field, enter 10.
17. In the Memory field, leave the value 128 MB.
18. Create a service account to be used for the function to process data from the device:
    1. Click Create an account.
    2. In the Create service account window that opens, enter an account name in the Name field. For example, my-db-function-service-account.
    3. Add the serverless.functions.invoker and editor roles for invoking functions and editing resources:
       1. Click the icon.
       2. Select a role from the list.
       3. Click Create.
19. Configure Environment variables. You can view database connection parameters in Yandex Managed Service for PostgreSQL.

    1. Click Add environment variable.

    2. Fill in the Key and Value fields for the environment variables:

       Key	Description	Value
       VERBOSE_LOG	Turning data logging on and off.	True
       DB_HOSTNAME	Hostname in Yandex Managed Service for PostgreSQL.	See the management console for Yandex Managed Service for PostgreSQL
       DB_PORT	Port to connect to the cluster in Yandex Managed Service for PostgreSQL.	6432
       DB_NAME	Cluster name in Yandex Managed Service for PostgreSQL.	db1
       DB_USER	Username to connect to the cluster in Yandex Managed Service for PostgreSQL.	user1
       DB_PASSWORD	Password to connect to the cluster in Yandex Managed Service for PostgreSQL.	The password you set in Yandex Managed Service for PostgreSQL.

20. In the upper-right part of the window, click Create version.

Test the data processing function

To test the function:

1. In management console, select the folder where you are running the scenario.

2. Select Cloud Functions.

3. In the left part of the window, select Testing.

4. In the Version tag field, select $latest (most recent function).

5. In the Payload field, insert the following data:

   {
       "messages": [
           {
               "event_metadata": {
                   "event_id": "160d239876d9714800",
                   "event_type": "yandex.cloud.events.iot.IoTMessage",
                   "created_at": "2020-05-08T19:16:21.267616072Z",
                   "folder_id": "b112345678910"
               },
               "details": {
                   "registry_id": "are1234567890",
                   "device_id": "are0987654321",
                   "mqtt_topic": "$devices/are0987654321/events",
                   "payload": "ewogICAgICAgICAgICAiRGV2aWNlSWQiOiJhcmU1NzBrZTA1N29pcjg1bDlmciIsCiAgICAgICAgICAgICJUaW1lU3RhbXAiOiIyMDIwLTA2LTExVDExOjA3OjIwWiIsCiAgICAgICAgICAgICJWYWx1ZXMiOlsKICAgICAgICAgICAgICAgIHsiVHlwZSI6IkJvb2wiLCJOYW1lIjoiU2VydmljZSBkb29yIHNlbnNvciIsIlZhbHVlIjoiRmFsc2UifSwKICAgICAgICAgICAgICAgIHsiVHlwZSI6IkZsb2F0IiwiTmFtZSI6IlBvd2VyIFZvbHRhZ2UiLCJWYWx1ZSI6IjI1LjA2In0sCiAgICAgICAgICAgICAgICB7IlR5cGUiOiJGbG9hdCIsIk5hbWUiOiJUZW1wZXJhdHVyZSIsIlZhbHVlIjoiMTEuMjEifSwKICAgICAgICAgICAgICAgIHsiVHlwZSI6IkZsb2F0IiwiTmFtZSI6IkNhc2ggZHJhd2VyIGZ1bGxuZXNzIiwiVmFsdWUiOiI2Ny44OSJ9LAogICAgICAgICAgICAgICAgeyJJdGVtcyI6WwogICAgICAgICAgICAgICAgICAgIHsiVHlwZSI6IkZsb2F0IiwgIklkIjoiMSIsIk5hbWUiOiJJdGVtIDEiLCJGdWxsbmVzcyI6IjUwLjY1In0sCiAgICAgICAgICAgICAgICAgICAgeyJUeXBlIjoiRmxvYXQiLCAiSWQiOiIyIiwiTmFtZSI6Ikl0ZW0gMiIsIkZ1bGxuZXNzIjoiODAuOTcifSwKICAgICAgICAgICAgICAgICAgICB7IlR5cGUiOiJGbG9hdCIsICJJZCI6IjMiLCJOYW1lIjoiSXRlbSAzIiwiRnVsbG5lc3MiOiIzMC4zMyJ9LAogICAgICAgICAgICAgICAgICAgIHsiVHlwZSI6IkZsb2F0IiwgIklkIjoiNCIsIk5hbWUiOiJJdGVtIDQiLCJGdWxsbmVzcyI6IjE1LjE1In0KICAgICAgICAgICAgICAgIF19CiAgICAgICAgICAgICAgICBdCiAgICAgICAgICAgIH0="
               }
           }
       ]
   }
   

6. Click Run test.

If the function is successful, the Function status field shows Done and the Function output field shows the following:

{
"statusCode" : 200 ,
    "headers" : {
	    "Content-Type" : "text/plain"
    },
"isBase64Encoded" : false
}

View the data processing result in Managed Service for PostgreSQL

To view the data processing result:

1. In management console, select the folder where you are running the scenario.
2. Select Yandex Managed Service for PostgreSQL.
3. Go to the SQL section.

A table with data processing results is displayed in the right part of the window.

Create a trigger to invoke the data processing function

The trigger invokes the function when the device topic gets messages.

To create a trigger:

1. In management console, select the folder where you are running the scenario.
2. Select Cloud Functions.
3. Select Triggers.
4. Click Create trigger.
5. In the Name field, enter a name for the trigger. For example, my-db-func-trigger.
6. (optional) In the Description field, add additional information about the trigger.
7. Select a Type: Yandex IoT Core.
8. Under Yandex IoT Core message settings, enter the parameters previously set for the registry and device:

   • Registry: my-registry.

   • Device: Any device.

   • MQTT topic: $devices/#.

     Learn more about MQTT topics in Yandex IoT Core.

9. Under Function settings, enter the parameters previously set for the function:
   • Function: my-database-function.
   • Function version tag: $latest.
   • Service account: my-db-func-trigger-service-account.
10. (optional) Set parameters under Repeat request settings and Dead Letter Queue settings. They provide data security.
    • Repeat request settings are useful if you want to recall the function when the current function request fails.
    • Dead Letter Queue settings are necessary to forward messages that consumers couldn't process in standard queues.
      You can configure a standard message queue as a DLQ. If you haven't created a message queue yet, create one in Yandex Message Queue.
11. Click Create trigger.

View the trigger result in Managed Service for PostgreSQL

Shortly after creating the trigger, you can check how it works.

To view the trigger result:

1. In management console, select the folder where you are running the scenario.
2. Select Yandex Managed Service for PostgreSQL.
3. Go to the SQL section.

In the right part of the window, you'll see an updated table with a large amount of data.

Configure monitoring in Yandex DataLens

Set up a connection to Managed Service for PostgreSQL

To set up a Yandex DataLens connection to Managed Service for PostgreSQL:

1. In management console, select the folder where you are running the scenario.

2. Select Yandex DataLens.

3. Click Create connection.

4. Select the PostgreSQL connector.

   When connecting to an external data source (which is not a Yandex.Cloud resource), grant access to the source for the DataLens IP address ranges.

5. Enter a name for the connection: MyPGConnection.

6. Select a Connection: Select in cloud.

7. Under Cluster, select my-pg-database.

8. Under Hostname, select the host you created in Yandex Managed Service for PostgreSQL.

9. In the Port field, enter 6432.

10. Under Database name, select db1.

11. Under Username, select user1.

12. In the Password field, enter the password you set to access the cluster in Yandex Managed Service for PostgreSQL.

13. Click Create. The connection appears in the list.

Create a dataset

To create a dataset:

1. In management console, select the folder where you are running the scenario.
2. Select Yandex DataLens.
3. Click Create dataset.
4. To the left of the screen, click ** Add **.
5. Select MyPGConnection.
6. In the left part of the window, select the public.iot_events and public.iot_position tables, and drag them to the right.
7. Click Save.
8. In the window that opens, set the name of the dataset to My-pg-dataset and click Create.

The dataset appears in the list.

Create a chart for temperature and mains voltage

To create a chart for temperature and mains voltage:

1. In management console, select the folder where you are running the scenario.

2. Select Yandex DataLens.

3. Click Create chart.

4. Under Dataset, select My-pg-dataset, which you created earlier.

5. Select the Line chart chart type.

6. From the Dimensions section in the left part of the window, drag the dimension to the Line chart section:

   • event_datetime to the X section.

     A timeline will appear in the lower part of the chart on the X-axis.

   • temperature and power_voltage to the Y section.

     The temperature and mains voltage values will appear on the Y-axis of the chart.

7. Click Save.

8. In the window that opens, set the chart name or use the generated name My-pg-dataset — Line chart and click Save.

Create a chart with a map

To create a chart with a map:

1. In management console, select the folder where you are running the scenario.

2. Select Yandex DataLens.

3. In the left part of the window, select Datasets.

4. In the list, select My-pg-dataset.

5. Go to the Fields tab.

6. In the right part of the window, click Add field.

7. In the window that opens, in the Field name field, enter Position.

8. In the Formula field, insert GEOPOINT([latitude],[longitude]).

9. Click Create.

10. Click Save.

11. In the upper-right corner, click Create chart.

12. Select the Map chart type.

13. From the Dimensions section in the left part of the window, drag the measurements to the Map section:

    • Position to the Geopoints section.

    • item1_fullness, item1_fullness, item1_fullness, item1_fullness, cash_drawer, and coin_drawer to the Tooltips section.

      In the right part of the window, you'll see a scalable map with vending machines marked with points on the map and tooltips marked with legend lines.

14. Click Save.

15. In the window that opens, set the chart name or use the generated name My-pg-dataset — Map and click Save.

Create a dashboard

To create a dashboard:

1. In management console, select the folder where you are running the scenario.
2. Select Yandex DataLens.
3. Click Create dashboard.
4. In the window that opens, enter the dashboard name MyDash.
5. Add the following charts to the dashboard: My-pg-dataset — Map and My-pg-dataset — Line chart, which you created earlier:

   1. In the Add dropdown list at the top of the window, select Chart.

   2. In the dropdown Chart list, select the My-pg-dataset — Map chart.

      The chart name will appear in the Name field.

   3. (optional) In the Description field, enter a description for the chart.

   4. Click Add.

   5. Repeat the steps to add the My-pg-dataset — Line chart chart.

6. Configure the selector:
   1. In the Add dropdown list at the top of the window, select Selector.
   2. In the Name field, enter Device.
   3. In the Dataset list, select My-pg-dataset.
   4. In the Field list, select device_id.
   5. In the Default value list, select the ID of the device you created in Yandex IoT Core.
   6. Click Add.
7. Configure links:

   1. At the top of the window, click Links.

   2. In the list at the top of the window that opens, select the My-pg-dataset — Point map chart.

   3. In the dropdown In.link list, select Ignore.

      Maps don't support the Links parameter.

   4. Click Save.

   5. From the list at the top of the window, select the My-pg-dataset — Line chart chart.

   6. In the In.link dropdown list, select In.link.

   7. Click Save.

8. Click Save.

Learn more about Yandex DataLens dashboards.