Get started
Yandex Managed Service for Kubernetes

Configuring Fluent Bit for Yandex Cloud Logging

Written by

The Fluent Bit log processor lets you transfer the Managed Service for Kubernetes cluster logs to Yandex Cloud Logging. The Fluent Bit plugin for Yandex Cloud Logging module is used to transfer logs.

To set up transfer of logs:

  1. Prepare the Kubernetes cluster.
  2. Install and configure Fluent Bit.

Before you begin

  1. If you don't have the Yandex Cloud command line interface yet, install and initialize it.

    The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

  2. Create a service account, which is required for Fluent Bit.

    1. Assign it the role logging.writer.

    2. Create an authorized key and save it to a file named key.json:

      yc iam key create \
  --service-account-name <name of service account for Fluent Bit> \
  --output key.json

  3. Create a log group.

  4. Create a Kubernetes cluster and a node group in any suitable configuration with Kubernetes version 1.21 or higher.

  5. Install kubectl and configure it to work with the created cluster.

Install and configure Fluent Bit

  1. Install Fluent Bit by following the instructions.
  2. Check transmission of Kubernetes cluster logs to Cloud Logging.

  1. Create the objects necessary for Fluent Bit to run:

    • For clusters with Kubernetes versions 1.21 and lower:

      kubectl create namespace logging && \
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml && \
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml && \
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml

    • For clusters with Kubernetes versions 1.22 and higher:

      kubectl create namespace logging && \
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml && \
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-1.22.yaml && \
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding-1.22.yaml

  2. Create a secret including the key of the service account:

    kubectl create secret generic secret-key-json \
  --from-file=key.json \
  --namespace logging

  3. Download the config.yaml configuration file:

    wget https://raw.githubusercontent.com/knpsh/yc-logging-fluent-bit-example/main/config.yaml

  4. Enter the log output parameters in the data.output-elasticsearch.conf section in config.yaml:

    ...
  output-elasticsearch.conf: |
    [OUTPUT]
      Name            yc-logging
      Match           *
      group_id        <log group ID>
      resource_id     <optional: cluster ID Kubernetes>
      message_key     log
      authorization   iam-key-file:/etc/secret/key.json
...

    You can request the log group ID with a list of log groups in the folder.

    If necessary, specify additional settings for the Fluent Bit.

  5. Create Fluent Bit objects:

    kubectl apply -f config.yaml

    Result:

    configmap/fluent-bit-config created
daemonset.apps/fluent-bit created

  6. Make sure the Fluent Bit status changed to Running:

    kubectl get pods -n logging

  7. Check transmission of Kubernetes cluster logs to Cloud Logging.

Delete the resources you created

If you no longer need these resources, delete them:

  1. Delete the Managed Service for Kubernetes cluster.
  2. If you reserved a public static IP address for the cluster, delete it.
  3. Delete the log group.
In this article: