Yandex Cloud
  • Services
  • Solutions
  • Why Yandex Cloud
  • Blog
  • Pricing
  • Documentation
  • Contact us
Get started
Language / Region
Yandex project
© 2023 Yandex.Cloud LLC
Yandex Managed Service for Kubernetes
  • Comparison with other Yandex Cloud services
  • Getting started
  • Step-by-step instructions
    • All instructions
    • Connecting to a node over SSH
    • Updating Kubernetes
    • Configuring autoscaling
    • Connecting to a cluster
      • Connection method overview
      • Configuring security groups
      • Creating a static configuration file
    • Installing applications from Cloud Marketplace
      • Basics of working with Cloud Marketplace
      • Installing Argo CD
      • Installing Container Storage Interface for S3
      • Installing Crossplane
      • Installing External Secrets Operator
      • Installing Filebeat
      • Installing Fluent Bit
      • Installing Gateway API
      • Installing GitLab Agent
      • Installing GitLab Runner
      • Installing HashiCorp Vault
      • Installing the Application Load Balancer Ingress controller
      • Installing Jaeger
      • Installing Kyverno & Kyverno Policies
      • Installing Metrics Provider
      • Installing NodeLocal DNS
      • Installing Policy Reporter
      • Installing Thumbor
    • Network scenarios
      • Granting access to an app running in a Kubernetes cluster
      • Configuring the Calico network policy controller
      • Configuring the Cilium network policy controller
      • Configuring NodeLocal DNS for the Cilium network policy controller
    • Working with persistent volumes
      • Dynamic volume provisioning
      • Static volume provisioning
      • Managing storage classes
      • Expanding a pod volume
      • Expanding a StatefulSet controller volume
      • Mounting a volume in Block mode
      • Integration with Object Storage
    • Managing a Kubernetes cluster
      • Getting information about a Kubernetes cluster
      • Creating a Kubernetes cluster
      • Editing a Kubernetes cluster
      • Creating a namespace in a Kubernetes cluster
      • Deleting a Kubernetes cluster
    • Managing a node group
      • Information about existing node groups
      • Creating a node group
      • Connecting to a node over SSH
      • Configuring autoscaling
      • Changing a node group
      • Managing Kubernetes cluster node labels
      • Deleting a node group
    • Connecting external nodes to the cluster
  • Practical guidelines
    • All tutorials
    • Creating a new Kubernetes project in Yandex Cloud
    • Integration with Container Registry
    • Running workloads with GPUs
    • Installing an NGINX Ingress controller with Let's Encrypt®
    • Installing an NGINX Ingress controller with a certificate from Certificate Manager
    • Backups to Object Storage
    • Horizontal application scaling in a cluster
    • Vertical application scaling in a cluster
    • Deploying and load testing a gRPC service with scaling
    • Working with snapshots
    • Integrating into a corporate DNS zone
    • Automatic DNS scaling by cluster size
    • Setting up local DNS caching
    • Checking DNS Challenge for Let's Encrypt® certificates
    • Monitoring a cluster using Prometheus and Grafana
    • Continuous deployment of containerized applications using GitLab
    • Using Cloud Marketplace products
      • Integrating with Argo CD
      • Integration with Crossplane
      • Syncing with Yandex Lockbox secrets
      • Configuring Fluent Bit for Cloud Logging
      • Setting up Gateway API
      • Configuring the Application Load Balancer Ingress controller
      • Using Jaeger to trace requests in Managed Service for YDB
      • Using Metrics Provider to stream metrics
  • Concepts
    • Relationships between service resources
    • Release channels and updates
    • Encrypting secrets
    • Using Kubernetes API objects
      • Volumes
      • Service
    • Node groups
      • Cluster autoscaler
      • Evicting pods from nodes
      • Dynamic resource allocation for a node
      • Node groups with GPUs
    • Network in Managed Service for Kubernetes
    • External cluster nodes
    • Network settings and cluster policies
    • Automatic scaling
    • Quotas and limits
    • Recommendations for using Managed Service for Kubernetes
  • Access management
  • Pricing policy
  • API reference
    • Authentication in the API
    • gRPC
      • Overview
      • ClusterService
      • NodeGroupService
      • VersionService
      • OperationService
    • REST
      • Overview
      • Cluster
        • Overview
        • create
        • delete
        • get
        • list
        • listNodeGroups
        • listNodes
        • listOperations
        • start
        • stop
        • update
      • NodeGroup
        • Overview
        • create
        • delete
        • get
        • list
        • listNodes
        • listOperations
        • update
      • Version
        • Overview
        • list
  • Questions and answers
    • General questions
    • Data storage
    • Configuring and updating
    • Automatic scaling
    • Resources
    • Logs
    • All questions on one page
  1. Questions and answers
  2. Automatic scaling

Questions and answers about node group autoscaling in Managed Service for Kubernetes

Written by
Yandex Cloud

    Why are there N nodes in my cluster now, but it's not getting downsized?

    Autoscaling doesn't stop nodes with pods that can't be evicted. Scaling barriers:

    • Pods whose eviction is limited with PodDisruptionBudget.
    • Pods in the kube-system namespace:
      • That haven't been created under the DaemonSet controller.
      • That don't have PodDisruptionBudget or whose eviction is limited by PodDisruptionBudget.
    • Pods that haven't been created under a replication controller (ReplicaSet, Deployment, or StatefulSet).
    • Pods with a local storage.
    • Pods that can't be evicted anywhere due to restrictions. For example, due to lack of resources or lack of nodes matching the affinity or anti-affinity selectors.
    • Pods with an annotation that disables eviction: "cluster-autoscaler.kubernetes.io/safe-to-evict": "false".

    Note

    Pods kube-system, pods with local-storage, and pods without a replication controller can be evicted. To do this, set the annotation "safe-to-evict": "true":

    kubectl annotate pod <pod name> cluster-autoscaler.kubernetes.io/safe-to-evict=true
    

    Other possible causes:

    • The node group has already reached its minimum size.

    • The node is idle for less than 10 minutes.

    • During the last 10 minutes, the node group has been scaled up.

    • During the last 3 minutes, there was an unsuccessful attempt to scale down the node group.

    • There was an unsuccessful attempt to stop a certain node. In this case, the next attempt occurs in 5 minutes.

    • The node has an annotation that prohibits stopping it on scale-down: "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true". You can add or remove annotation using kubectl.

      Check for annotation on the node:

      kubectl describe node <node name> | grep scale-down-disabled
      

      Result:

      Annotations:        cluster-autoscaler.kubernetes.io/scale-down-disabled: true
      

      Set the annotation:

      kubectl annotate node <node name> cluster-autoscaler.kubernetes.io/scale-down-disabled=true
      

      Remove the annotation by running the kubectl command with -:

      kubectl annotate node <node name> cluster-autoscaler.kubernetes.io/scale-down-disabled-
      

    Why has the pod been deleted, but the node group fails to downsize?

    If the node is underloaded, it's removed in 10 minutes.

    Why isn't autoscaling performed even when the number of nodes gets less than the minimum or greater than the maximum?

    Autoscaling won't violate preset limits, but Managed Service for Kubernetes isn't explicitly controlling the limits. Scaling up will only trigger if there are pods in the unschedulable status.

    To get answers to other questions about autoscaling, see the documentation Kubernetes.

    Was the article helpful?

    Language / Region
    Yandex project
    © 2023 Yandex.Cloud LLC