Transmitting the Managed Service for Kubernetes cluster logs to Cloud Logging

The Fluent Bit log processor lets you transfer the Managed Service for Kubernetes cluster logs to Yandex Cloud Logging. The Fluent Bit plugin for Yandex Cloud Logging module is used to transfer logs.

To set up transfer of logs:

1. Prepare the Kubernetes cluster.
2. Install and configure Fluent Bit.

Before you start

1. If you don't have the Yandex Cloud command line interface yet, install and initialize it.

   The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

2. Create a service account needed for Fluent Bit to run.

   1. Assign it the role logging.writer.

   2. Create an authorized key and save it to key.json:

      yc iam key create \
        --service-account-name <name of service account for Fluent Bit> \
        --output key.json
      

3. Create a log group.

4. Create a Managed Service for Kubernetes cluster and a node group in any suitable configuration with the Kubernetes version 1.21 or higher.

5. Install kubectl and set it up for working with the created cluster.

Prepare the Kubernetes cluster

1. Create the objects necessary for Fluent Bit to run:

   kubectl create namespace logging && \
   kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml && \
   kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml && \
   kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml
   

2. Create a secret including the key of the service account:

   kubectl create secret generic secret-key-json \
     --from-file=key.json \
     --namespace logging
   

Install and configure Fluent Bit

1. Create a folder for the service account key and copy the key there:

   sudo mkdir -p /etc/secret && \
   sudo cp key.json /etc/secret
   

2. Download the config.yaml configuration file:

   wget https://raw.githubusercontent.com/knpsh/yc-logging-fluent-bit-example/main/config.yaml
   

3. Enter the log output parameters in the data.output-elasticsearch.conf section in config.yaml:

   ...
     output-elasticsearch.conf: |
       [OUTPUT]
         Name            yc-logging
         Match           *
         group_id        <log group ID>
         resource_id     <optional: Kubernetes cluster ID>
         message_key     log
         authorization   iam-key-file:/etc/secret/key.json
   ...
   

   You can request the log group ID with a list of log groups in the folder.

   If needed, enter additional settings for Fluent Bit.

4. Create Fluent Bit objects:

   kubectl apply -f config.yaml
   

   Expected execution result:

   configmap/fluent-bit-config created
   daemonset.apps/fluent-bit created
   

5. Make sure the Fluent Bit status changed to Running:

   kubectl get pods -n logging
   

6. Check the log transmission of the Kubernetes cluster to Cloud Logging.

Delete the resources you created

If you no longer need these resources, delete them:

1. Delete the cluster Managed Service for Kubernetes.
2. If you reserved a public static IP address for the cluster, delete it.
3. Delete the log group.