Fluent Bit with a Cloud Logging plugin

Fluent Bit is an open-source multi-platform tool for collecting and processing logs. Fluent Bit collects, parses, and filters messages from a variety of inputs and saves them to storage. Once a message is saved, it is forwarded from storage to a router that determines which destination to route it to. Plugins are used to work with various inputs and outputs.

A plugin for Cloud Logging is used to export logs generated in Kubernetes clusters, including at the pod level, to Yandex Cloud Logging.

1. Create a Kubernetes cluster and a node group.

2. Create a service account to run Fluent Bit.

3. Assign the service account the logging.writer role.

4. Create an authorized key for the service account and save it to a file named sa-key.json:

   yc iam key create \
      --service-account-name <service account name> \
      --output sa-key.json
   

5. Create a log group.

6. If you want to save logs to Yandex Object Storage:

   1. Create a bucket.

   2. Assign the service account the storage.uploader role.

   3. Create a static access key for the service account and save it in JSON format:

      {
          "access_key": {
          "id": "xxxxxx",
          "created_at": "xxxxxx",
          "key_id": "xxxxxx"
          },
          "secret": "xxxxxx"
      }
      

7. Set the Fluent Bit settings:

   • Namespace: Select a namespace or create a new one.
   • Application name: Enter an application name.
   • Log group ID: Specify an ID of the log group to which logs will be transmitted.
   • Service account key: Paste the contents of the sa-key.json file.
   • Save logs to Object Storage: Select this option if you want to save logs to Object Storage.
   • Object Storage bucket name: Specify the name of the bucket where logs will be stored. Leave the field blank if you do not want to save logs.
   • Static access key for Object Storage: Paste the static access key in JSON format. Leave the field blank if you do not want to save logs.

8. Click Install.

• Exporting Kubernetes cluster logs to Yandex Cloud Logging.

Yandex Cloud technical support responds to requests 24/7. The types of requests available and their response time depend on your pricing plan. You can activate paid support in the management console. Learn more about requesting technical support.