- Sensitive data
- Resource model
- Network security Managed Service for Kubernetes
- Authentication and access control Managed Service for Kubernetes
- Secure Managed Service for Kubernetes configuration
- Data encryption and Managed Service for Kubernetes secret management
- Protection against malicious code in Managed Service for Kubernetes
- Vulnerability management Managed Service for Kubernetes
- Security updates
- Backup and recovery
- Security policies in Kubernetes
- Practices for securely creating and using Docker images
- Runtime protection
- Load sharing between nodes
- Collecting, monitoring, and analyzing audit logs Managed Service for Kubernetes
- Kubernetes security solutions comparison
The section includes recommendations for Yandex Cloud users on security settings in Yandex Managed Service for Kubernetes.
The user is responsible for all actions made inside the Kubernetes node. The user is responsible for the security of the nodes and their proper setup in accordance with PCI DSS requirements and other security standards.
Yandex Cloud is responsible for the the Kubernetes API security.
The user is responsible for correctly choosing security settings in Managed Service for Kubernetes, including selecting the channel and the update schedule.
When using Managed Service for Kubernetes to comply with PCI DSS or other security standards, it is forbidden to:
- Use sensitive data in names and descriptions of clusters, node groups, namespaces, services, and pods.
- Use sensitive data in Kubernetes node labels and Yandex Cloud service resource labels.
- Use sensitive data in pod manifests.
- Use sensitive data in etcd in clear text.
- Write sensitive data to Managed Service for Kubernetes logs.
Wherever possible, ensure maximum isolation between resources:
- Use a separate organization for each "large-scale" project.
- Use a separate cloud for each development team.
- Use a separate Kubernetes cluster located in a separate folder for each service.
- Use a separate namespace for each microservice.
Your clouds must have no shared resources. Cloud members must have access only to their clouds.
Less strong isolation models are also possible, for example:
- Projects are split between different clouds.
- Development teams are assigned independent folders.
- Services have separate Kubernetes clusters.
- Microservices have independent namespaces.
Network security Managed Service for Kubernetes
We don't recommend that you grant access to the Kubernetes API and node groups from non-trusted networks (for example, from the internet).
Use firewall protection when needed (for example, security groups). In the section below, you can find links to instructions on how to set up firewall protection in security groups.
Restrict network access to the Kubernetes API (master) and node groups using instructions for security groups.
When using an ALB as an Ingress Gateway, also complete the following steps:
Apply the security group to the ALB.
Additionally, apply the security group to the node group:
- Source type:
<security group applied to the ALB>.
- Destination type:
- Port range: 30000-32767.
- Source type:
Restrict network access inside Kubernetes using the Network Policy.
You can use two network plugins in Yandex Cloud:
- Calico: A basic plugin.
- Cilium CNI: An advanced plugin that uses advanced network policies applied at the L7 layer (REST/HTTP, gRPC and Kafka).
We recommend that you use the
default deny rule for the default incoming and outgoing traffic, allowing only relevant traffic.
To generate policies, you can use the Cilium CNI built-in Hubble platform to analyze the traffic manually. Various solutions for automatic generation of network policies are also available on the market.
For useful examples of network policies, see the repo.
A helpful tool to create both basic and advanced network policies is available here.
Setting up incoming network access
For online endpoints, we recommend that you allocate an independent Kubernetes cluster or independent node groups (using Taints and Tolerations + Node affinity mechanisms). By doing this, you establish a DMZ so that if your nodes are compromised online, your attack surface is small.
To enable incoming network access to your workloads via HTTP/HTTPS, use the Ingress resource.
There exist at least two variants of an Ingress controller that you can use in Yandex Cloud:
Benefits of Application Load Balancer Ingress controller:
- Integration with the Yandex Certificate Manager cloud service.
- No need to install a controller to the cluster because everything is deployed on the Application Load Balancer side).
Restricting access to the metadata of VMs in the node group
For all pods, create a network policy to block network traffic to port 169.254.169.254 or use the default-deny policy from the example. The policy must block workload node group metadata access because these node groups contain sensitive data, such as the token of the service account assigned to the node.
Authentication and access control Managed Service for Kubernetes
The access of IAM accounts to Managed Service for Kubernetes resources is managed at the following levels:
Managed Service for Kubernetesservice roles (access to the Yandex Cloud API): They enable you to control clusters and node groups (for example, create a cluster, create/edit/delete a node group, and so on).
Service roles to access the Kubernetes API: They let you control cluster resources via the Kubernetes API (for example, perform standard actions with Kubernetes: create, delete, view namespaces, work with pods, deployments, create roles, and so on). Only the basic global roles at the cluster level are available:
Primitive roles: These are global primitive IAM roles that include service roles (for example, the primitive role admin includes both the service administration role and the administrative role to access the Kubernetes API).
Standard Kubernetes roles: Inside the Kubernetes cluster, you can use Kubernetes tools to create both regular roles and cluster roles. This way, you can control IAM accounts access at the namespace level. To assign IAM roles at the namespace level, you can manually create RoleBinding objects in a relevant namespace, specifying the IAM ID of the cloud user in the "subjects name" field. Example:
apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: iam-user-aje0jndkhkvu04ek #name of the RoleBinding object namespace: micro1-ns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: admin subjects: - kind: User name: aje0jndkq855llvu04ek #cloud user ID
For the Managed Service for Kubernetes cluster to run, you need two service accounts: the service account of the cluster and the service account of the node group.
Secure Managed Service for Kubernetes configuration
In Managed Service for Kubernetes, the user is fully in control of all node group settings, but only partially in control of the master settings. These settings are part of the user's overall cluster security responsibility.
The CIS Kubernetes Benchmark standard is designed to build a secure Kubernetes configuration, including node configurations.
In Yandex Cloud, the Kubernetes node groups are deployed by default with the configuration that complies with CIS Kubernetes Benchmark.
The kube-bench tool enables you to check whether the node group configuration is compliant with CIS Kubernetes Benchmark. The tool officially supports the Yandex Cloud node groups.
Here you can see the examples of launching kube-bench on the nodes.
In addition, kube-bench supports integration with Starboard Operator — another product for kube-bench automatic launching.
Starboard Operator is a free tool that helps you automate scanning of images for vulnerabilities and checking that the configuration complies with CIS Kubernetes Benchmark.
Integrity control (FIM — File integrity monitoring)
You must control two levels of file integrity in node groups:
- OS files of the node - for example, configuration files.
- Container files - for example, critical files that the user application writes to the volume.
OS files of the node
A comprehensive solution in Osquery and kubequery in K8s.
One of the methods to solve this task:
- Use readOnlyRootFilesystem in pods.
- Make sure to mount folders to write the data to as separate volumes: as emptydir or individual disks.
If you mount folders as emptydir, files are stored on the node in the folder
/var/lib/kubelet/pods/PODUID/volumes/kubernetes.ioempty-dir/VOLUMENAME. To ensure data integrity, you can monitor this folder by Osquery as OS node files.
In the case of separate disks (not emptydir), you can mount volumes in read mode to the above-mentioned DaemonSet running Osquery.
To control file integrity on the Kubernetes nodes, you can also use the tools listed in Integrity control.
There exist dedicated free solutions for Kubernetes nodes from Google or Argus, including file-integrity-operator.
Data encryption and Managed Service for Kubernetes secret management
At the Kubernetes etcd level, encrypt secrets using an in-built mechanism from Yandex Cloud.
We recommend that you use SecretManager solutions to work with Kubernetes secrets. Yandex Lockbox is such a solution in Yandex Cloud.
Yandex Lockbox was integrated with Kubernetes using the External Secrets open-source project. The solution is available in Cloud Marketplace in the basic simplified scenario: External Secrets Operator with Yandex Lockbox support.
Useful instructions on working with External Secrets:
- Instructions to work with External Secrets and Yandex Lockbox from the project description.
- Instructions to work with External Secrets and Yandex Lockbox from the Yandex Cloud documentation.
Many methods to differentiate access to secrets using this tool have been described.
The most secure recommended option for encrypting secrets is ESO as a Service (External Secrets Operator as a service). In this case, the global administrator has access to the namespace where ESO is installed, and administrators of specific namespaces create their respective
SecretStore objects (where they specify IAM authorized access keys for their Lockbox secrets). If this
SecretStore object is compromised, only the authorized key of one specific namespace is compromised (rather than all of them, as in the case of Shared ClusterSecretStore).
Encryption in transit
For in-transit encryption, use TLS interaction between pods. If you can't use TLS interaction, use service mesh solutions:
Encryption at rest
If you need to encrypt your stored data, you can use:
- Key Management Service, for encrypting data at the application level (including when you use persistent volumes).
- A custom method of data encryption. However, in this case, protection of the keys and of the key management procedure is the sole responsibility of the user.
Protection against malicious code in Managed Service for Kubernetes
There are two levels where you can enable malicious code protection in Kubernetes:
- Container Registry level protection.
- OS-level protection of Kubernetes nodes.
Security scanner in Container Registry.
To protect the containerization host levels, you can use a variety of paid and free solutions from the "Runtime security" and "Antivirus engine" classes. Examples of free solutions:
Vulnerability management Managed Service for Kubernetes
Yandex Cloud within Managed Service for Kubernetes is in charge of vulnerability management and security updates on the master. The user must independently control vulnerabilities on the Kubernetes worker nodes.
Scanning for vulnerabilities
You can break vulnerability scanning into the following levels:
- Image-level vulnerability scanning.
- Vulnerability scanning of the OS nodes in Kubernetes.
Vulnerability scanning at the image level is detailed in Protection against malicious code in Managed Service for Kubernetes.
Examples of free universal solutions for vulnerability scanning of the OS nodes in Kubernetes are given in Scanning for vulnerabilities.
There also exist both paid and free solutions for scanning the OS nodes in Kubernetes and Kubernetes hosts for vulnerabilities: for example, free tools such as kube-hunter and trivi (scan filesystem).
Managed Service for Kubernetes issues updates in a regular manner. To meet the Information Security standards:
- Select a relevant update channel and enable either automatic installation of updates, or manual installation immediately after publication in the selected channel.
- Double-check that the ad settings meet the Information Security standards.
- Use one of the three latest Kubernetes versions, because updates (including security updates) are only released for these versions.
Backup and recovery
Security policies in Kubernetes
Requirements listed in Pod Security Standards from Kubernetes let you prevent threats related to Kubernetes objects.
To implement the requirements, you can either use the Kubernetes built-in Pod Security Admission Controller tool or open-source software (for example, other Admission Controllers: OPA Gatekeeper, Kyverno).
Examples using Kyverno:
- Analyzing Kubernetes security logs in ELK: audit logs, Policy Engine, Falco.
- Example of setting up role models and policies in Managed Service for Kubernetes.
To control compliance with Pod Security Standards, you can also use the following tools within CI/CD:
Or a separate Kubesec tool.
Practices for securely creating and using Docker images
Use these check lists to meet requirements for secure creation of images:
You can control Dockerfile in your CI/CD pipeline using the Conftest utility.
When using minimal images or distroless images without a shell, use ephemeral containers.
Load sharing between nodes
Data loads with different security contexts (such as different severities of data processed) must be processed on different Kubernetes nodes. To enable load sharing within a cluster, use different node groups with different settings for
node labels and
node taints. Be sure to combine both settings.
Collecting, monitoring, and analyzing audit logs Managed Service for Kubernetes
Events available to the user in the Managed Service for Kubernetes service can be classified as levels:
- Kubernetes API events (Kubernetes Audit logging).
- Kubernetes node events.
- Kubernetes pod events.
- Kubernetes metrics.
- Kubernetes Flow logs.
Kubernetes API level (Kubernetes Audit logging).
Audit events are collected from the Kubernetes API level by Cloud Logging.
Kubernetes node level
Kubernetes node level events are collected and exported similarly to collecting OS audit logs.
Kubernetes pod level
Different options for collecting and exporting pod-level events in Kubernetes is described in the Kubernetes official documentation.
Examples of collecting and exporting pod logs:
- Exporting logs to Cloud Logging using Fluent Bit is described in the Managed Service for Kubernetes documentation.
- Exporting pod logs into Elastic or Splunk is described in the Yandex Cloud Security Solution Library.
Yandex Monitoring includes a set of metrics to analyze availability of Kubernetes objects and their behavioral anomalies.
Instructions on how to export Yandex Monitoring metrics is given in the section Exporting events to SIEM.
Kubernetes Flow logs
Managed Service for Kubernetes role model audit
In the Managed Service for Kubernetes console, you can audit the current role model used in the service. For this, go to the Access management tab in the service.
You can also use: