Kubernetes

“Is this a controller or an operator?” comes up in every platform team sooner or later, usually when someone proposes writing one. The honest answer is that every operator is a controller, but not every controller is an operator. The line between them is not technical. It is about what the loop knows.

A controller is a loop that watches Kubernetes objects and moves the real state of the cluster toward the desired state written in those objects. An operator is a controller that manages a specific application through its own custom resources and carries the operational knowledge a human operator would otherwise have: how to install it, upgrade it, back it up, fail it over and recover it.

This article goes through the definitions precisely, compares the two in a table, shows what a minimal reconcile loop looks like, covers the frameworks you would use in 2026, and, maybe most usefully, lists the cases where you should build neither. If you want the gentler introduction to the operator pattern first, start with Kubernetes operators: the basics and come back.

What a Kubernetes Controller Is

Kubernetes is built on one idea: you declare the state you want, and something keeps making it true. That “something” is a controller.

A controller runs a reconciliation loop:

1. Watch one or more resource types through the API server.
2. When something changes, read the desired state (the object’s spec).
3. Observe the actual state (other objects, or the outside world).
4. Act to close the gap: create, update or delete things.
5. Write what it observed into the object’s status.
6. Repeat, forever.

Two properties make this model work.

Controllers are level-triggered, not edge-triggered. A controller does not react to “a pod was deleted” as an event to process exactly once. It reacts to “the world may have changed, go look”. If it misses an event, restarts, or receives five changes at once, the next reconcile still computes the right answer, because it compares full desired state against full actual state. This is why reconcile functions must be idempotent: running them twice in a row must be harmless.

Controllers own a narrow slice of the world. The Deployment controller manages ReplicaSets. The ReplicaSet controller manages Pods. The scheduler assigns Pods to nodes. The kubelet runs containers. None of them does the others’ job; the behavior you see when you kubectl apply a Deployment is the emergent result of several small loops cooperating through the API server.

The built-in controllers

Most of the controllers you use every day live in a single binary, kube-controller-manager, which runs on the control plane. It bundles dozens of loops, among them:

• Deployment and ReplicaSet controllers, which roll out and maintain pod replicas.
• StatefulSet, DaemonSet, Job and CronJob controllers.
• The EndpointSlice controller, which keeps Service backends up to date.
• The Namespace controller, which cleans up everything when a namespace is deleted.
• The garbage collector, which deletes dependents through ownerReferences.
• The node lifecycle controller, which marks nodes unhealthy and evicts pods.
• The horizontal pod autoscaler controller, which adjusts replica counts from metrics (see Kubernetes HPA best practices for how that loop behaves).
• The PersistentVolume binder and expansion controllers.

Cloud-specific loops (load balancers, node addresses, routes) run separately in the cloud-controller-manager. And plenty of third-party components are controllers too: ingress controllers, the ArgoCD application controller, external-dns, cluster-autoscaler, Karpenter.

None of those is usually called an operator. They are generic infrastructure: they manage Kubernetes primitives or a single external concern, without encoding how to run a particular piece of software.

What a Kubernetes Operator Is

The term “operator” was coined by CoreOS in 2016 to describe a specific use of the controller pattern: putting the knowledge of a human operator into software. The official Kubernetes documentation defines operators as “software extensions to Kubernetes that make use of custom resources to manage applications and their components”.

An operator has three ingredients:

1. A Custom Resource Definition (CRD) that models the application in its own vocabulary. You do not create a StatefulSet, a Service, a ConfigMap, a PodDisruptionBudget and a CronJob for backups. You create a PostgresCluster or a Kafka object with fields like instances, version and backup.schedule.
2. A controller that reconciles those custom resources into the underlying Kubernetes objects and external state.
3. Domain knowledge baked into that controller: the day-2 operations that are specific to the application. Promoting a replica when the primary dies. Running a schema migration before rolling out a new version. Rebalancing partitions after scaling a broker set. Taking a consistent backup. Rotating certificates without downtime.

The third ingredient is the one that matters. A controller that turns a WebApp custom resource into a Deployment plus a Service has a CRD and a reconcile loop, so it technically fits the definition, but it knows nothing a Helm chart does not know. A PostgreSQL operator that notices the primary is unreachable, picks the most up-to-date replica, promotes it, re-points the read-write Service and re-attaches the old primary as a replica when it comes back is doing something no template can do.

Controller vs Operator: Side-by-Side Comparison

In one sentence: the controller is the pattern, the operator is a use of the pattern for managing an application with custom resources and embedded operational expertise.

The Grey Zone: Is cert-manager an Operator?

The boundary is fuzzy, and arguing about it is mostly unproductive, but a few examples help calibrate.

cert-manager introduces CRDs (Certificate, Issuer, ClusterIssuer, CertificateRequest) and reconciles them. It encodes real domain knowledge: the ACME protocol, HTTP-01 and DNS-01 challenges, renewal windows, key rotation. Yet its own documentation calls it a controller, and it does not manage an application in the “install and run this database” sense. It manages a concern: certificates. Most people would call it an operator anyway, and that is fine.

ArgoCD has CRDs (Application, ApplicationSet, AppProject) and an application controller. It is a controller for GitOps delivery, not an operator for a specific workload. There is, however, an ArgoCD Operator that installs and upgrades ArgoCD itself, which shows how the word tends to be used: “the thing that runs X for you”. (More on ArgoCD itself in the ArgoCD guide.)

A custom WebApp CRD that expands into a Deployment, Service and Ingress is a controller with a CRD. Calling it an operator is not wrong, but it is an operator with capability level 1, which leads to the next section.

Operator Capability Levels

The Operator Framework defines a five-level maturity model that is the most useful way to tell a “real” operator from a templating controller:

A level 1 operator does roughly what helm install does. The reason operators exist is levels 2 and 3. If your design stops at level 1, you probably want a Helm chart, not an operator.

A Minimal Reconcile Loop in Go

The most common way to write a controller or operator in Go is controller-runtime, the library under Kubebuilder and Operator SDK. Here is the shape of a reconcile function for a hypothetical Website custom resource that owns a Deployment:

func (r *WebsiteReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    // 1. Read the desired state.
    var site webv1.Website
    if err := r.Get(ctx, req.NamespacedName, &site); err != nil {
        // Deleted between the event and now: nothing to do.
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 2. Build the object we want to exist.
    deploy := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{Name: site.Name, Namespace: site.Namespace},
    }

    // 3. Create or update it idempotently, owned by the Website.
    op, err := controllerutil.CreateOrUpdate(ctx, r.Client, deploy, func() error {
        deploy.Spec.Replicas = ptr.To(site.Spec.Replicas)
        deploy.Spec.Selector = &metav1.LabelSelector{MatchLabels: map[string]string{"app": site.Name}}
        deploy.Spec.Template.Labels = map[string]string{"app": site.Name}
        deploy.Spec.Template.Spec.Containers = []corev1.Container{{
            Name:  "web",
            Image: site.Spec.Image,
        }}
        return controllerutil.SetControllerReference(&site, deploy, r.Scheme)
    })
    if err != nil {
        return ctrl.Result{}, err // controller-runtime requeues with backoff
    }
    log.Info("deployment reconciled", "operation", op)

    // 4. Report observed state.
    site.Status.ReadyReplicas = deploy.Status.ReadyReplicas
    site.Status.ObservedGeneration = site.Generation
    if err := r.Status().Update(ctx, &site); err != nil {
        return ctrl.Result{}, err
    }
    return ctrl.Result{}, nil
}

func (r *WebsiteReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&webv1.Website{}).     // reconcile when a Website changes
        Owns(&appsv1.Deployment{}). // ...or when a Deployment it owns changes
        Complete(r)
}

Notice what the function does not do. It never asks “what changed?”. It reads the whole desired state, computes the whole target object and lets CreateOrUpdate make it so. The owner reference means that deleting the Website garbage-collects the Deployment, and Owns() means a manual edit to the Deployment triggers a reconcile that puts it back.

This is a controller with a CRD. To turn it into an operator worth the name, the reconcile function would grow the domain logic: detecting that spec.version moved from 3 to 4 and running a migration Job first, holding the rollout until the Job succeeds, setting a Degraded condition if it fails, taking a backup before any destructive change. That logic, not the scaffolding, is where the months of work go.

A few mechanics every non-trivial operator ends up needing:

• Finalizers, so the controller can clean up external resources (a cloud bucket, a DNS record, a database user) before the custom resource disappears.
• Status conditions following the standard type/status/reason/message shape, plus observedGeneration, so users and tools can tell whether the latest spec has been processed.
• Requeue with a delay (ctrl.Result{RequeueAfter: ...}) for things that must be polled, such as an external API that does not emit Kubernetes events.
• Leader election, so only one replica of the operator acts at a time.
• Narrow RBAC and namespace-scoped watches where possible, because an operator is a highly privileged component.

Frameworks for Writing Controllers and Operators in 2026

You almost never write the watch, cache and work-queue machinery yourself. The main options, all actively maintained as of September 2026:

controller-runtime and Kubebuilder (Go)

controller-runtime (kubernetes-sigs, v0.25 at the time of writing) is the library: manager, cached client, informers, work queues, leader election, webhooks. Kubebuilder (v4.16) is the scaffolding and code generator on top: it creates the project layout, CRD types with markers, RBAC manifests, webhooks and an envtest-based test setup. If you write Go and have no special constraints, this is the default choice, and the one most operators in the CNCF landscape use.

Operator SDK (Go, Helm, Ansible)

Operator SDK (Operator Framework, v1.42) uses Kubebuilder’s plugins for Go projects, so the Go experience is nearly identical. What it adds is integration with the Operator Lifecycle Manager (OLM) for packaging and distributing operators as bundles, a scorecard for testing against best practices, and two non-Go flavors: Helm-based operators, which reconcile a custom resource by rendering a chart, and Ansible-based operators, which run playbooks. Choose it if you target OpenShift or OperatorHub, or if your team would rather write Ansible than Go.

Java Operator SDK (Java)

Java Operator SDK (v5.6, released September 2026) brings the same model to the JVM, with a Quarkus extension that produces native images. Recent releases added informer sharing between controllers and a built-in event recorder. It is the natural fit for teams whose domain logic and client libraries already live in Java; Strimzi and Keycloak’s operators are prominent JVM examples of the pattern.

kopf (Python)

kopf (Kubernetes Operator Pythonic Framework, 1.44) lets you write handlers as decorated Python functions, such as @kopf.on.create('websites'). It is excellent for glue operators, prototypes and teams of SREs who live in Python. It is less common for large, performance-sensitive operators, but a lot of internal automation runs happily on it.

Metacontroller (any language, via webhooks)

Metacontroller (v4.17) inverts the model: it runs the watch-and-reconcile machinery for you and calls your webhook with the parent object and its current children, and your webhook returns the desired children. Your “controller” is a stateless HTTP function in any language. It is ideal for simple composite resources and decorators, and a poor fit for operators with complex multi-step state machines.

Declarative alternatives: kro and Crossplane

If what you want is “one custom resource that expands into a bundle of other resources”, you may not need to write a controller at all. kro (Kube Resource Orchestrator, a Kubernetes SIG Cloud Provider subproject) lets you define a ResourceGraphDefinition that generates a new CRD and its reconciliation from a declarative graph; note that its APIs are still pre-1.0 and not declared production-ready. Crossplane compositions do something similar, with a focus on cloud infrastructure. Both give you a level 1 “operator” without Go code.

When to Write a Controller, an Operator, or Neither

This is the decision that matters, and the default answer should be “neither”.

Use a Helm chart (or Kustomize) when the application is stateless or its state is managed elsewhere, installation is the hard part, and upgrades are “roll out a new image”. Templates plus helm upgrade plus a sane rollout strategy cover the vast majority of internal services. Chart hooks handle the occasional pre-upgrade migration (see the Helm hooks guide), and there are well-known patterns for organizing charts across many services (centralized vs per-service chart management).

Use an existing operator when you are running well-known stateful software. PostgreSQL, Kafka, Redis, Elasticsearch, RabbitMQ and Prometheus all have mature operators whose authors have already hit the failure modes you have not. Writing your own PostgreSQL operator to avoid learning CloudNativePG is a multi-year mistake.

Write a plain controller when you need to keep something continuously in sync and no existing tool does it: labeling namespaces from an external CMDB, creating DNS records or cloud IAM bindings from annotations, enforcing a cleanup policy. There may be no CRD at all; the controller can watch built-in resources and act on annotations.

Write an operator when all of these are true:

• The software has stateful day-2 operations that humans currently perform by runbook: failover, resharding, ordered upgrades, backup and restore.
• Those operations are frequent or risky enough that automating them pays back the investment.
• No mature operator already exists, or the existing ones are unmaintained or cannot be adapted.
• You have the team to own it long-term. An operator is production software with its own release cycle, API versioning, conversion webhooks and CVEs.

Real-World Operators Worth Studying

Reading good operators is the fastest way to understand the gap between a controller and an operator.

Prometheus Operator manages Prometheus and Alertmanager through CRDs such as Prometheus, Alertmanager, ServiceMonitor, PodMonitor and PrometheusRule. Its key insight is turning scrape configuration into Kubernetes objects that each team can own next to its application; the operator regenerates Prometheus configuration and reloads it. If you have used kube-prometheus-stack, you have used it. The distinction between its two main discovery CRDs is covered in ServiceMonitor vs PodMonitor.

CloudNativePG runs PostgreSQL with a Cluster CRD and manages primary and replica pods directly rather than through a StatefulSet, precisely because it needs finer control over which instance is primary, how failover happens and how volumes are handled. It covers switchover, automated failover, continuous backup and point-in-time recovery to object storage, and rolling minor-version upgrades. It is a textbook level 3+ operator.

Strimzi runs Apache Kafka with a cluster operator plus topic and user operators, modeling brokers, node pools, topics, users and connectors as custom resources. It handles rolling restarts in the order Kafka needs, certificate management between brokers and clients, and scaling with partition reassignment through Cruise Control.

cert-manager, discussed above, is the canonical example of a controller-shaped project with deep domain knowledge of a protocol rather than of a workload.

The Logging Operator is another good example of a CRD-driven design where teams declare log flows and outputs per namespace, and the operator manages the Fluent Bit and Fluentd deployments behind them.

Anti-Patterns

The common ways operators go wrong:

An operator that just runs Helm. A controller whose reconcile function renders a chart and applies it adds a CRD, a Deployment, RBAC and a new failure mode, and gives you nothing helm upgrade in a GitOps pipeline would not. If there is no day-2 logic, delete it.

Edge-triggered thinking. Writing reconcile as “on create do A, on update do B” leads to drift the first time an event is missed. Always compute from the full desired state.

Non-idempotent side effects. Sending an email, charging a card or creating a cloud resource without first checking whether it already exists will happen twice. Record what you did in status or use deterministic names.

No status. An operator that never writes conditions leaves users running kubectl logs on the operator pod to guess what happened. Status is the operator’s user interface.

Cluster-admin RBAC. Operators are high-value targets. Request only the verbs and resources you use, and prefer namespace-scoped installs where the software allows it.

Hot requeue loops. Returning an error or Requeue: true on every reconcile for a condition that will not change for minutes hammers the API server. Use RequeueAfter with a sensible interval.

Blocking finalizers. A finalizer that can never complete (because the external system is gone) leaves objects stuck in Terminating forever. Always have a timeout or an escape hatch.

CRD versioning as an afterthought. Once users have objects stored as v1alpha1, changing the schema means conversion webhooks and storage migration. Design the API as carefully as a public REST API, because it is one.

Frequently Asked Questions

Conclusion

The controller-versus-operator debate is less about definitions than about intent. A controller is the fundamental Kubernetes building block: a level-triggered, idempotent loop that keeps some part of the world matching a declared state. An operator is what you get when you point that building block at one application, give it a vocabulary through custom resources, and teach it everything your on-call engineers know about running that application.

That makes the practical guidance simple. Use Helm for installation, reach for a mature existing operator for well-known stateful systems, write a small controller when you need continuous synchronization that nothing else provides, and write a full operator only when there is real day-2 knowledge to encode and a team ready to own it for years. If you are about to build a level 1 operator, you are probably about to build a Helm chart with extra steps.

Yes — Kubernetes can change the CPU and memory of a running pod without deleting it and, in most cases, without restarting its containers. The feature is called In-Place Pod Resize (the KEP calls it In-Place Pod Vertical Scaling), it has been stable since Kubernetes 1.35, and on any cluster at 1.33 or later it is already on by default. You trigger it by patching the pod’s resize subresource:

kubectl patch pod checkout-7d9f8c6b5-x2lqp --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"1"},"limits":{"cpu":"2"}}}]}}'

The kubelet updates the container’s cgroup, the pod keeps its name, IP, node and in-memory state, and restartCount stays where it was. That is the short answer. The long answer — which this article is about — is that “without restarting” comes with conditions: which resource you are changing, what resizePolicy says, whether the node has room, what QoS class the pod was born with, and whether the runtime inside the container can actually use the new numbers. And the moment you want this to happen automatically, you are in Vertical Pod Autoscaler territory, where the new InPlaceOrRecreate and InPlace modes finally change the old “VPA means evictions” trade-off — and where running HPA and VPA together still needs care.

If you need the fundamentals first, the VPA introduction covers when vertical scaling makes sense at all, and Kubernetes resource requests and limits covers what the numbers you are about to change actually do.

Why Resource Changes Used to Mean a Restart

For most of Kubernetes’ history, spec.containers[*].resources was immutable on a Pod. Requests and limits were decided at admission, the scheduler placed the pod against them, the kubelet wrote them into the container’s cgroup, and that was it for the life of the pod. If you wanted more CPU, you changed the Deployment template, the Deployment controller created a new ReplicaSet, and the old pods were terminated one by one.

That is fine for a stateless web tier with twenty replicas and a fast readiness probe. It is painful for:

• stateful workloads where a restart means cache warm-up or a replica resync;
• long-running batch or ML jobs where a restart throws away hours of progress;
• JVM services that need a lot of CPU for thirty seconds of startup and very little afterwards.

It is also why the Vertical Pod Autoscaler had a reputation problem. Its only way to apply a new recommendation to a running pod was to evict it and let the admission webhook stamp the new values on the replacement. Many teams ran VPA in recommendation-only mode forever because they did not want an autoscaler that deletes pods.

Feature History: From Alpha to Stable

In-Place Pod Resize took more than six years from conception to GA. The version history matters because behavior changed along the way, and a lot of blog posts still describe the alpha or beta semantics.

On 1.35+ the feature cannot be turned off, and your kubectl client needs to be 1.32 or newer to use --subresource resize; older clients fail with an “invalid subresource” error even against a new API server.

How In-Place Resize Works

Three fields carry the whole mechanism, and reading them correctly saves most of the debugging time.

spec.containers[*].resources is the desired state. For CPU and memory, these fields are now mutable — but only through the resize subresource. A normal kubectl edit pod or kubectl apply against the main pod resource is still rejected for resource changes.

status.containerStatuses[*].resources is the actual state. It reflects what the kubelet has really configured in the container’s cgroup. When desired and actual differ, a resize is pending or in progress.

status.containerStatuses[*].allocatedResources is what the kubelet has admitted. It is mostly for scheduling. While a resize is incomplete, the scheduler accounts for the pod using the maximum of desired, allocated and actual requests, so a node never gets overcommitted in the gap.

The flow is:

1. You (or an autoscaler) patch the pod’s resize subresource with new requests and limits.
2. The API server validates the change — QoS class preserved, only CPU and memory touched, no resource removed.
3. The kubelet on the pod’s node checks whether the new requests fit in the node’s allocatable capacity.
4. If they fit, the kubelet admits the resize, updates the cgroup through the container runtime (containerd or CRI-O via the CRI UpdateContainerResources call), and — depending on resizePolicy — restarts the container or not.
5. When the runtime confirms, status.containerStatuses[*].resources matches the spec and the conditions clear.

A Complete Example

Create a pod with an explicit resize policy:

apiVersion: v1
kind: Pod
metadata:
  name: resize-demo
  namespace: resize-lab
spec:
  containers:
    - name: app
      image: registry.k8s.io/pause:3.9
      resizePolicy:
        - resourceName: cpu
          restartPolicy: NotRequired      # default, explicit here
        - resourceName: memory
          restartPolicy: RestartContainer
      resources:
        requests:
          cpu: 500m
          memory: 256Mi
        limits:
          cpu: "1"
          memory: 512Mi

kubectl create namespace resize-lab
kubectl apply -f resize-demo.yaml

This pod is Burstable (requests differ from limits). Now double the CPU:

kubectl -n resize-lab patch pod resize-demo --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"1"},"limits":{"cpu":"2"}}}]}}'

Check desired versus actual, and the restart count:

kubectl -n resize-lab get pod resize-demo \
  -o jsonpath='{.spec.containers[0].resources}{"\n"}{.status.containerStatuses[0].resources}{"\n"}{.status.containerStatuses[0].restartCount}{"\n"}'

Both resource blocks should show cpu: 1 / cpu: 2, and restartCount is still 0. Now change the memory limit:

kubectl -n resize-lab patch pod resize-demo --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app","resources":{"limits":{"memory":"768Mi"}}}]}}'

This time restartCount goes to 1, because the memory policy is RestartContainer. The pod is the same pod — same UID, same IP, same node — but the container process was restarted to pick up the new value.

resizePolicy: NotRequired vs RestartContainer

resizePolicy is a per-container, per-resource setting with two values:

• NotRequired (the default when unset): apply the new value to the running container’s cgroup without restarting it.
• RestartContainer: restart the container so the process starts with the new value.

The rule when several resources change at once is simple: if any changed resource has RestartContainer, the container restarts. With the policy above, a CPU-only change is in place, a memory-only change restarts, and a CPU-and-memory change restarts.

One restriction: if the pod’s overall restartPolicy is Never (typical for Jobs that must not rerun a container), every resizePolicy entry must be NotRequired. You cannot declare a resize that would require a restart the pod is not allowed to do.

Which policy should you choose?

CPU: almost always NotRequired. Changing CPU requests and limits changes CFS shares and quota in the cgroup. The kernel applies it on the next scheduling period and the process does not need to know. The one caveat is runtimes that size thread pools from the CPU count at startup — the JVM’s GC threads and ForkJoinPool.commonPool() are the classic example. The process gets more CPU time immediately, but it will not create more worker threads. (Go is the pleasant exception: since Go 1.25 the runtime reads the cgroup CPU limit and adjusts GOMAXPROCS while running, unless you pinned it with an environment variable.) For most services that is fine; for heavily parallel compute, a restart may still be worth it.

Memory: depends on whether the process can use it. Raising a cgroup memory limit is harmless, but a JVM started with -XX:MaxRAMPercentage=75 computed its max heap from the old limit and will not grow the heap past it. Python and most interpreted runtimes do not re-read limits either — the Kubernetes GA announcement itself notes that Java and Python do not support resizing memory without a restart. For these, RestartContainer on memory is the honest setting: you still avoid rescheduling and keep the pod’s identity, you just restart the process. Go with GOMEMLIMIT set from the downward API, or native services that allocate on demand, can use NotRequired.

Memory Decreases: Allowed, but Best-Effort

Until 1.35, decreasing a container’s memory limit in place was simply forbidden. The GA release lifted that restriction with a safeguard: when the memory restartPolicy is NotRequired, the kubelet checks current memory usage before applying a lower limit. If usage is already above the new limit, the resize is not applied and stays in progress rather than triggering an immediate OOM kill.

The check is explicitly best-effort. There is a race: usage can spike between the check and the cgroup write, and then the kernel OOM-kills the container. Treat memory decreases on NotRequired containers as something you do with margin — shrink to well above the observed working set, not to it. Moving the check into the container runtime to close that race is listed as future work.

Watching a Resize: Conditions, Deferred and Infeasible

When the kubelet cannot apply a resize immediately, it tells you through pod conditions.

PodResizePending means the kubelet has not admitted the resize. The reason field says why:

• Infeasible: the request can never fit on this node — for example, asking for 16 CPUs on an 8-CPU node, or a change the node’s configuration forbids. It will not be retried until you change the request.
• Deferred: the request does not fit right now, but could if other pods leave. The kubelet retries automatically.

PodResizeInProgress means the kubelet has admitted the resize and allocated the resources but the runtime has not finished applying them. It is usually brief. If actuation fails, the condition carries reason: Error and a message.

To see them:

kubectl -n resize-lab get pod resize-demo \
  -o jsonpath='{range .status.conditions[?(@.type=="PodResizePending")]}{.reason}{": "}{.message}{"\n"}{end}'

kubectl -n resize-lab describe pod resize-demo | sed -n '/Conditions:/,/Volumes:/p'

Since 1.35 the pod also emits dedicated events for resizes, and status.observedGeneration tells you which version of the spec the kubelet has acknowledged — useful when several resizes are issued in a row and you need to know which one the status refers to.

When several resizes on a node are Deferred, they are retried in priority order: higher PriorityClass first, then Guaranteed before Burstable, then the longest-waiting first. A deferred high-priority resize does not block lower-priority ones that happen to fit. And with the 1.37 alpha InPlacePodVerticalScalingSchedulerPreemption gate, the scheduler can go further and evict lower-priority pods to make room for a higher-priority pod’s deferred resize — the same preemption semantics as scheduling, applied to growth.

Limitations You Will Hit

As of Kubernetes 1.37, the official list of restrictions is:

The other restriction that is not in the table but matters more than all of them: in-place resize is a Pod operation, not a Deployment operation. If you change resources in a Deployment, StatefulSet or DaemonSet template, the controller still performs a normal rolling update with new pods. Workload controllers do not propagate template changes in place. Resizing in place means patching individual pods — by hand, with a script, or with an autoscaler that does it for you. And whatever you patch onto a pod is lost the next time the controller replaces it, because new pods are created from the template.

Pod-Level Resources Resize

Kubernetes also supports resources at the pod level — spec.resources as an aggregate budget shared by all containers — and since 1.36 those can be resized in place too (beta, on by default; alpha in 1.35 under InPlacePodLevelResourcesVerticalScaling). The same resize subresource, conditions and retry logic apply:

kubectl patch pod pod-level-demo --subresource resize --patch \
  '{"spec":{"resources":{"requests":{"cpu":"300m"},"limits":{"cpu":"300m"}}}}'

Two rules are specific to pod-level resize. The new pod-level requests must be at least the sum of the container requests, and no individual container limit may exceed the new pod-level limit. And there is no pod-level resizePolicy: the pod cgroup is always changed in place, but containers without their own limit inherit the pod limit as their effective ceiling, so their container-level resizePolicy still decides whether they restart. A container with restartPolicy: RestartContainer for CPU and no CPU limit will restart when you raise the pod’s CPU limit.

Automating It: VPA InPlaceOrRecreate and InPlace

Manual resizes are useful for incidents and experiments. The production use of this feature is the Vertical Pod Autoscaler, which as of VPA 1.8 has six update modes:

InPlaceOrRecreate is the mode most teams should move to:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  updatePolicy:
    updateMode: InPlaceOrRecreate
  resourcePolicy:
    containerPolicies:
      - containerName: app
        controlledResources: ["cpu", "memory"]
        minAllowed:
          cpu: 250m
          memory: 256Mi
        maxAllowed:
          cpu: "4"
          memory: 4Gi

The VPA updater attempts an in-place update when the pod’s requests fall outside the recommended bounds, after a quick OOM, or for pods running more than 12 hours whose recommendation differs by more than 10%. It falls back to eviction when the resize is Infeasible, has been Deferred for more than 5 minutes, has been in progress for more than an hour, would change the pod’s QoS class, or requires a memory limit decrease the policy does not allow. All containers of a pod are updated together.

A few operational details worth knowing:

• In-place updates still respect VPA’s eviction tolerance and minReplicas by default. If every container uses NotRequired for both resources, the updater flag --in-place-skip-disruption-budget lets VPA skip those checks for updates that genuinely do not restart anything.
• The updater exports vpa_updater_in_place_updated_pods_total and vpa_updater_failed_in_place_update_attempts_total, among others. Alert on the failure counter — a steady stream of failed in-place attempts usually means nodes are too full and you are silently back to evictions.
• InPlace mode is for workloads where any eviction is unacceptable. The price is that a resize that never fits never happens; VPA tracks infeasible attempts and only retries when the recommendation drops below what was already rejected.

CPU Startup Boost: The JVM Use Case

The most compelling use of in-place resize is the startup spike. A Spring Boot or Quarkus-on-JVM service can burn two or three cores for the first 30–60 seconds (class loading, JIT compilation, connection pool warm-up) and then settle at 200 millicores. Sizing requests for the startup means paying for idle cores all day; sizing for steady state means slow starts, failing startup probes and an HPA that adds pods too late to help.

VPA 1.7 introduced CPU Startup Boost (alpha, CPUStartupBoost feature gate), built directly on in-place resize:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders
  updatePolicy:
    updateMode: InPlaceOrRecreate
  startupBoost:
    cpu:
      type: Factor
      factor: 3
      durationSeconds: 60

The admission controller multiplies the CPU of new pods by the factor, and once the pod is Ready and durationSeconds has elapsed, the updater scales CPU back down in place — to the VPA recommendation if VPA controls the container, otherwise to the original spec. VPA 1.8 added reactive unboosting on top. Boost can also be set per container under containerPolicies, or as a fixed Quantity instead of a factor.

Remember the caveat from the resizePolicy section: the JVM sizes GC and common pool threads at startup, so it keeps the thread counts it chose during the boost — usually a benefit, not a problem.

Using HPA and VPA Together Without Conflicts

“Can I use HPA and VPA together?” is the other question people search for, and in-place resize changes the answer only partially.

The classic conflict: an HPA with a CPU or memory utilization target computes utilization as usage divided by requests. VPA changes requests. When VPA raises the CPU request, utilization drops, and the HPA scales in; with fewer pods, usage per pod rises, VPA raises requests again. The two controllers fight over the same signal from opposite directions. In-place resize makes that loop faster, not safer, because VPA no longer has to wait for evictions to take effect.

The rule remains: never let HPA and VPA act on the same resource metric. The patterns that work:

HPA on a custom or external metric, VPA on resources. Scale out on requests per second, queue depth or consumer lag (via the custom metrics API, Prometheus Adapter or KEDA), and let VPA InPlaceOrRecreate right-size CPU and memory per pod. Each controller owns a different dimension. This is the pattern I recommend for most services that need both.

HPA on CPU, VPA on memory only. Set controlledResources: ["memory"] in the VPA’s container policy. HPA owns CPU-driven horizontal scale, VPA only sizes memory. Works well for services whose memory footprint depends on configuration and data size rather than load.

HPA on CPU, VPA in Off mode. Use VPA purely as a recommendation engine and feed its numbers into the Deployment template through your GitOps pipeline. No live conflict, and you still get data-driven requests.

If your HPA scales on CPU and you are also tuning its behavior, the HPA best practices guide and the HPA memory examples cover the scaling policies. For HPA on event-driven metrics, see KEDA and event-driven autoscaling.

Interaction With Cluster Autoscaling

In-place resize never moves a pod. If a node is full, a resize goes Deferred, and neither Cluster Autoscaler nor Karpenter will add a node for it: they react to unschedulable pods, and a pod with a deferred resize is scheduled and running. The pod stays at its old size until something on the node shrinks or leaves.

That is why InPlaceOrRecreate falls back to eviction after five minutes deferred: an evicted pod with larger requests becomes pending, and pending pods do trigger node provisioning. If you run VPA in pure InPlace mode, plan headroom on nodes explicitly, or accept that some resizes will wait. The 1.37 preemption alpha helps high-priority pods but does not create capacity. For how node autoscalers make those decisions, see Cluster Autoscaler vs Karpenter, and if you are adopting several of these newer features at once, Kubernetes features that hurt in production is a useful sanity check on rollout order.

Frequently Asked Questions

Conclusion

In-place pod resize turns CPU and memory from birth certificates into dials. On Kubernetes 1.35+ you can grow or shrink a running pod through the resize subresource, and for CPU that genuinely means no restart. Memory is more nuanced: the cgroup changes live, but many runtimes only read their memory budget at startup, so RestartContainer is often the right policy — still far cheaper than rescheduling.

The bigger shift is for autoscaling. VPA’s InPlaceOrRecreate mode, GA since VPA 1.6, removes the main reason teams kept VPA in recommendation-only mode, and CPU Startup Boost finally gives JVM services a clean answer to the startup spike. What has not changed is the rule for combining autoscalers: give HPA and VPA different signals, keep the QoS class in mind from day one, and remember that anything you patch onto a pod lives only until the next rollout.

On September 24, 2026 I shut down the three MicroK8s VMs that had run my homelab for years and switched every workload over to a new Talos Linux cluster on the same Proxmox hosts. The cluster runs my GitOps-managed apps: observability, n8n, analytics, a few side projects, and the tools I use to write this blog. It has been in production for a few days now, including a rolling reboot of all three nodes.

The move itself was not hard. Talos is well documented, and talosctl is one of the better CLIs in the Kubernetes ecosystem. What took time were the small things that MicroK8s had been doing for me quietly, which I only noticed when they stopped: which IP the kubelet advertises, how CoreDNS resolves my own domains, which Ceph messenger protocol the kernel client speaks, and why a snapshot backup of a VM can freeze etcd.

This post covers the architecture I ended up with, the config I generated, and a checklist of every gotcha, so you can skip the evenings I spent on them. If you have never looked at Talos before, start with my Talos Linux guide. If you are still choosing an immutable OS, my Talos vs Bottlerocket vs Flatcar comparison covers that decision.

Why I Left MicroK8s

MicroK8s served me well. It is a single snap: snap install microk8s, enable a few addons, join the nodes, and you have a cluster. For a homelab that is hard to beat, and I would still recommend it to anyone who wants Kubernetes on one Ubuntu machine by this afternoon.

Over time, three problems added up:

The node OS was one more thing to look after. Every MicroK8s node was a full Ubuntu VM with SSH, apt updates, unattended-upgrades, snap refreshes and the odd kernel that needed a reboot. My package update script handled it, but every VM was a pet with its own drift.

Snap refreshes were something I had to plan around. Pinning a channel works, but the addons (DNS, ingress, MetalLB, hostpath storage) are MicroK8s’ own versions of those components, with their own defaults. When something broke, I was debugging MicroK8s’ packaging on top of Kubernetes itself.

Recovery was not declarative. If a node died, rebuilding it meant a long runbook: install Ubuntu, install the snap, enable the same addons, rejoin. Talos turns that into “apply this YAML to a fresh VM”.

Talos takes the opposite approach: no SSH, no shell, no package manager, a read-only root filesystem, and one API (talosctl) through which you configure everything. The whole node is a machine config file. For a homelab that I want to forget about between upgrades, that was the deciding factor.

Target Architecture

The Proxmox cluster has three physical hosts with Ceph running across them. The Talos layout mirrors that:

Three control-plane nodes that also run workloads is a deliberate homelab trade-off. Etcd gets quorum and survives losing one host, and I don’t spend RAM on dedicated workers that would sit mostly idle. In a production cluster I would split them. Here, removing the NoSchedule taint (allowSchedulingOnControlPlanes: true in the cluster config) is the pragmatic choice.

The VIP is one of my favourite Talos features. You declare a shared IP on the control-plane network interface, and Talos runs the election between the nodes: whichever node holds etcd leadership answers on the VIP. No keepalived, no HAProxy VM, no external load balancer for the API. The kubeconfig points at the VIP, and my reverse proxy points ingress traffic at the same address.

Generating the Machine Config

Talos config generation is one command plus patches. I keep the cluster secrets (the CA, the etcd CA, and the bootstrap tokens) in my password manager, not in git, and generate everything else from versioned patch files:

talosctl gen config homelab https://<VIP>:6443 \
  --with-secrets secrets.yaml \
  --config-patch @machine/patch-common.yaml \
  -o base

talosctl machineconfig patch base/controlplane.yaml \
  -p @machine/patch-node1.yaml \
  -p @machine/patch-nodeip.yaml \
  -p @machine/patch-metrics.yaml \
  -p @machine/patch-apiserver-sans.yaml \
  -p @machine/patch-etcd-tuning.yaml \
  -o node1.yaml

Each patch has one job:

• patch-common: settings shared by every node (VIP, scheduling on control planes, and so on).
• patch-nodeN: hostname and static addresses for that specific node.
• patch-nodeip: which subnet the kubelet and etcd may use as the node’s address. This one matters more than you would think (next section).
• patch-metrics: exposes controller-manager, scheduler and etcd metrics to Prometheus.
• patch-apiserver-sans: extra names in the API server certificate so the DNS name and the VIP both validate.
• patch-etcd-tuning: etcd settings adjusted for VMs whose disks are shared with everything else on the host.

Keeping secrets.yaml out of the repo while versioning the patches means I can rebuild any node’s config from scratch in under a minute. Without the secrets file, a regenerated config would create a brand-new cluster identity that the existing nodes reject.

Why the nodeIP patch is mandatory with two networks

This was my first real surprise. Each VM has two NICs: the LAN and the Ceph network. With no guidance, the kubelet picked the Ceph network address as the node’s InternalIP on some nodes. Everything looked healthy (pods scheduled, kubectl get nodes was all green), but NodePorts and the host-network ingress were only listening on the Ceph network, which my reverse proxy cannot reach.

The fix is to tell the kubelet which subnet is valid. While you’re at it, pin etcd’s advertised subnet too:

machine:
  kubelet:
    nodeIP:
      validSubnets:
        - 192.168.1.0/24
cluster:
  etcd:
    advertisedSubnets:
      - 192.168.1.0/24

If your nodes have more than one interface, apply this patch before you bootstrap. Changing etcd’s advertised address after the fact is a lot more painful than setting it up front.

The CPU Type Gotcha: x86-64-v3 for Ceph CSI

My VMs started with Proxmox’s default CPU model, and Talos booted and ran fine. Then the ceph-csi pods went into CrashLoopBackOff with no useful log output. The current ceph-csi images are built for a newer x86-64 microarchitecture level, and the default virtual CPU does not advertise the instructions they need.

Switching the VM CPU type to x86-64-v3 (or host, if all your Proxmox hosts have the same CPU generation and you don’t care about live migration between different CPUs) fixed it immediately. Set it when you create the VMs. Changing it later means a VM restart, which with three control-plane nodes is fine, but it is one more rolling operation you could have avoided.

Ceph CSI: RBD and CephFS Against Proxmox Ceph

I did not want a second storage system, so the cluster uses Proxmox’s existing Ceph through two Helm releases: ceph-csi-rbd and ceph-csi-cephfs. I deliberately kept the same pools and the same StorageClass names as on MicroK8s, with CephFS as the default class. That way the manifests in my GitOps repo work unchanged on both clusters, which made moving apps one by one much easier.

Two notes on this setup.

The CSI secrets are not in git. Each driver needs a Ceph user key: one each for the RBD provisioner, RBD node plugin, CephFS provisioner and CephFS node plugin. I create them out of band with ceph auth get-key client.<id> and a kubectl create secret. Sealing them would also work. I simply prefer that Ceph admin material never passes through the repo, even encrypted.

CephFS needs the v1 messenger port. RBD happily talks to the monitors on port 3300 (messenger v2). CephFS did not. Mounts failed with this error:

mount error: no mds is up or the cluster is laggy

The MDS was perfectly healthy. The problem was that the kernel CephFS client does not negotiate msgr v2 unless you pass the ms_mode mount option. With v2-only monitor addresses, it simply could not talk to the cluster. Listing the monitors on port 6789 (msgr v1) in the CephFS driver config fixed it. RBD kept using 3300.

If you see “no mds is up” and ceph -s says the MDS is active, check the monitor ports before you touch the MDS.

Migrating Persistent Volumes Without Copying Data

This was the part I was most nervous about, and it turned out to be the most elegant. Both clusters talk to the same Ceph pools, so the RBD images holding the data are already where they need to be. The only question is how to get the new cluster to use them.

The answer is static PersistentVolumes that reuse the original volumeHandle. The procedure for each app:

1. On MicroK8s, set the PV’s persistentVolumeReclaimPolicy to Retain, so deleting the claim never deletes the image.
2. Take a Ceph snapshot of the image as a safety net.
3. Scale the app to zero on MicroK8s.
4. Wait until rbd status <pool>/<image> shows no watchers. That confirms the old node has released the image.
5. In the GitOps repo, add a static PV on Talos with the same volumeHandle, the same CSI driver and Retain, plus a PVC bound to it by name.
6. Let ArgoCD sync the app on Talos.

A trimmed static PV looks like this:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: n8n-data
spec:
  capacity:
    storage: 10Gi
  accessModes: ["ReadWriteOnce"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ceph-rbd
  claimRef:
    namespace: n8n
    name: n8n-data
  csi:
    driver: rbd.csi.ceph.com
    fsType: ext4
    volumeHandle: <original-volume-handle-from-microk8s>
    volumeAttributes:
      clusterID: <ceph-fsid>
      pool: <pool>
      imageFeatures: layering
      staticVolume: "true"
    nodeStageSecretRef:
      name: csi-rbd-secret
      namespace: ceph-csi-rbd

Step 4 is not optional. My RBD images do not have the exclusive-lock feature, so nothing stops two clusters from mapping the same image at the same time. That is a textbook way to corrupt a filesystem. The same reasoning is why the MicroK8s VMs are now shut down with autostart disabled, and why they must stay that way: if they booted, they would try to remount volumes that Talos now owns.

One thing outside Kubernetes also bit me: my reverse proxy had a few custom locations with the old node IPs hardcoded. The main proxy target pointed at the new VIP, but a /api location on one host still went to a MicroK8s node. Search your proxy config for the old addresses before you shut anything down.

Ingress: Host Network, the public Class, and a 308 Loop

On MicroK8s, the ingress addon gives you an nginx controller with an IngressClass called public. All my Ingress manifests referenced it. Rather than rewriting every app, I installed ingress-nginx on Talos as a DaemonSet on the host network and named its IngressClass public as well. Ports 80 and 443 are open on all three nodes and therefore on the VIP, which is the only address my external reverse proxy needs to know.

Two gotchas came up.

Pod Security Admission blocks it by default. Talos enforces the baseline Pod Security Standard on every namespace. Host network and host ports violate it, so the controller pods were rejected until I labelled the namespace:

kubectl label namespace ingress \
  pod-security.kubernetes.io/enforce=privileged

Only do this on namespaces that really need it. This default is one of the things Talos gets right.

TLS terminated upstream causes a 308 redirect loop. My reverse proxy terminates TLS and forwards plain HTTP to the cluster. For any Ingress with a tls: section, ingress-nginx saw an HTTP request and answered with a 308 redirect to HTTPS. The proxy followed it, sent HTTP again, and so on. The fix is to make the controller trust the X-Forwarded-Proto header from the proxy:

controller:
  config:
    use-forwarded-headers: "true"

On a vanilla chart you have to say it explicitly.

I know ingress-nginx has been retired upstream, and it was a conscious choice to keep it for this migration. Changing the operating system, the storage path and the ingress controller in the same weekend is how you end up not knowing which change broke what. Gateway API is the next project. I wrote about how I approach that kind of move in lessons from the ingress-nginx migration, and the ingress-nginx alternatives post covers the options.

ArgoCD and Sealed Secrets: Reusing the Same Repo

I had one goal for GitOps: both clusters must be able to run from the same repository during the transition, so I could move apps one at a time and roll back by scaling down on one side and up on the other.

That turned out to be easy because every Application’s destination is https://kubernetes.default.svc, the in-cluster API, which means “whichever cluster this ArgoCD runs in”. I installed ArgoCD v3.3 from the official manifests on Talos, copied over argocd-cm, argocd-rbac-cm, argocd-secret and the repository credentials, and applied the same Application manifests. Apps that were still scaled to zero on MicroK8s synced normally on Talos. If you are new to ArgoCD, my ArgoCD guide covers the concepts I’m relying on here.

Sealed secrets need the old keys. Every SealedSecret in the repo was encrypted with the MicroK8s controller’s keys. A fresh controller on Talos would generate new keys and fail to decrypt all of them. I exported the existing key secrets (there were five, because the controller rotates keys), applied them in kube-system on Talos, and then installed the controller. It picked them up and decrypted everything on first sync.

The rotation deserves attention. sealed-secrets generates a new key every 30 days by default. Your offline backup of the keys therefore goes stale month by month. If you ever restore from it, anything sealed with a newer key will not decrypt. Re-exporting the keys periodically is now part of my runbook. For more on how secrets end up exposed (or not) inside a cluster, see discovering secrets in Kubernetes.

Manual Endpoints, because ArgoCD excludes them. My argocd-cm has a resource.exclusions rule for Endpoints and EndpointSlice, which is common and sensible: they’re usually generated by Kubernetes and would otherwise flood ArgoCD with noise. The side effect is that ArgoCD also ignores the hand-written Endpoints for Services without a selector. I use those to point Prometheus at targets outside the cluster (the NAS, an image proxy) and at control-plane components. After the migration, those ServiceMonitors had no targets. I now apply those Endpoints manually, and they’re documented in the cluster README so future me doesn’t lose another hour.

Control-Plane Metrics on Talos

kube-prometheus-stack expects to scrape the controller-manager, the scheduler and etcd. On Talos those components bind to localhost by default, so the targets show as down. The metrics patch fixes that:

cluster:
  controllerManager:
    extraArgs:
      bind-address: 0.0.0.0
  scheduler:
    extraArgs:
      bind-address: 0.0.0.0
  etcd:
    extraArgs:
      listen-metrics-urls: http://0.0.0.0:2381

There is a subtlety. Talos applied the controller-manager and scheduler changes live, but etcd only picked up the new metrics listener after a node reboot. For a while two of three etcd targets were up and one was down, which looked like a network problem until I realised only the rebooted node had the change. If you apply this patch to a running cluster, plan a rolling reboot.

CoreDNS: talosctl upgrade-k8s Overwrites Your Changes

MicroK8s’ DNS addon had a rewrite that sent all of my *.<my-domain> names to the reverse proxy’s LAN address. I had forgotten it existed. On Talos, one app could not reach another service through its public hostname, because my local DNS resolves that name to a VPN address that pods cannot route to.

The fix was a small hosts block in the CoreDNS Corefile that maps that hostname to the reverse proxy’s LAN IP. The catch: talosctl upgrade-k8s regenerates the CoreDNS ConfigMap, so the next Kubernetes upgrade would quietly remove the change. I now keep the live Corefile in the repo, and re-applying it is a step in my upgrade runbook. The cleaner long-term fix is to disable Talos’ built-in CoreDNS management and manage CoreDNS yourself with Helm, but for one hosts entry the runbook step is enough.

Backups: Don’t Snapshot Talos VMs, Snapshot etcd

This was the scariest incident of the migration. My Proxmox backup job ran in snapshot mode on the new Talos VMs like on every other VM. During the backup, the guest’s disk writes stalled long enough for etcd to start missing heartbeats. The API server became unresponsive, and the nodes did not recover cleanly until I rebooted them.

The lesson is that Talos nodes have no state worth backing up at the VM level. The OS is an immutable image, and the config is a YAML file you can regenerate. The only state that matters is etcd (plus the persistent volumes, which live in Ceph and have their own snapshots). So:

• The Talos VMs are excluded from the Proxmox backup job.
• A script on one of the Proxmox hosts runs talosctl etcd snapshot every night and keeps 14 days of snapshots on the NAS.
• A Prometheus alert fires if the snapshots stop appearing.

Restoring is one command against a freshly installed control-plane node:

talosctl -n <node> bootstrap --recover-from=./etcd-snapshot.db

Rehearse that restore on a throwaway VM before you need it. A backup you have never restored is only a hope.

MicroK8s vs Talos: How They Compare in Practice

The biggest change in daily work is debugging without SSH. My first reaction was to look for a way in, and there isn’t one. A few days in, I don’t miss it. talosctl dmesg, talosctl logs kubelet, talosctl dashboard and kubectl debug node/<name> (for the rare case where you need tools on the host) cover everything I have needed so far.

Gotchas Checklist

If you’re planning the same move, this is the list I wish I’d had:

• Set nodeIP.validSubnets (and etcd advertisedSubnets) before bootstrap if your VMs have more than one NIC.
• Use CPU type x86-64-v3 or newer on the Proxmox VMs, or ceph-csi will crash-loop.
• Point CephFS at the monitors on port 6789 (msgr v1) unless you set ms_mode. “no mds is up” is usually a protocol problem, not an MDS problem.
• Keep StorageClass and IngressClass names identical to the old cluster so manifests work in both during the transition.
• Migrate RBD volumes with static PVs reusing the volumeHandle, with Retain on both sides, and wait for zero watchers before mounting on the new cluster.
• Keep the old cluster off for good once its volumes have moved. Disable autostart.
• Grep your reverse proxy for the old node IPs, including custom locations.
• Label the ingress namespace privileged if the controller uses host networking.
• Enable use-forwarded-headers when TLS terminates in front of ingress-nginx.
• Copy the sealed-secrets keys before installing the controller, and re-export them after each 30-day rotation.
• Re-create hand-written Endpoints if ArgoCD excludes Endpoints from sync.
• Plan a rolling reboot after enabling etcd metrics.
• Keep your CoreDNS customisations in git and re-apply them after every talosctl upgrade-k8s.
• Exclude Talos VMs from VM-level snapshot backups and back up etcd with talosctl etcd snapshot instead. Rehearse the restore.
• Store secrets.yaml safely: it is the cluster’s identity. The rest of the config can be regenerated.

Frequently Asked Questions

Conclusion

Moving from MicroK8s to Talos Linux on Proxmox took a weekend of real work and a few evenings of gotchas. Almost none of those gotchas were Talos bugs. They were assumptions from the old cluster: which interface is the node’s address, what DNS rewrites exist, which Ceph protocol a client speaks, and whether a VM backup is harmless. Talos makes those assumptions visible because it does nothing you haven’t declared.

The result is a cluster where every node is a YAML file and a boot image, upgrades are a single talosctl command, and the only thing I really need to back up is etcd. Next up is replacing ingress-nginx with Gateway API, now that the platform underneath is no longer the variable.

If you’re planning a similar migration, start with the checklist above, keep both clusters able to run from the same GitOps repo, and move one app at a time.

kubectl exec stops working exactly when you need it most: the image has no shell, the container is stuck in CrashLoopBackOff, or the problem lives on the node rather than inside the pod. kubectl debug is the command built for those cases. It can attach a throwaway debug container to a pod that is already running, create a modified copy of a pod you can poke at safely, or drop you into a shell with the node’s filesystem mounted.

Related reading: OrbStack vs Docker Desktop: running containers and Kubernetes on a Mac.

This guide covers all three modes, the --profile flag (whose default changed in kubectl 1.36), custom profiles, the RBAC and Pod Security rules that usually get in the way, and the limitations nobody mentions until you hit them, such as the fact that you cannot remove an ephemeral container once it is added. If your specific problem is a distroless image, the deep dive on debugging distroless containers goes further on that case; this article is the general reference for the command itself.

What kubectl debug Does (and Why exec Is Not Enough)

kubectl exec runs a new process inside an existing container. That has three hard requirements: the container must be running, the binary you want (usually sh) must exist in its image, and whatever tools you need must also be in that image. Modern production images break all three. Distroless and scratch images ship no shell, crashing containers are never running long enough to exec into, and nobody wants tcpdump or strace baked into an application image.

kubectl debug removes those requirements by bringing its own image. Depending on the target, it does one of three things:

The command is available in every supported kubectl version. Ephemeral containers, the API feature behind the first mode, have been stable since Kubernetes 1.25, so any cluster you are likely to run today supports them.

Mode 1: Ephemeral Debug Containers in a Running Pod

This is the mode people mean when they search for “kubectl debug container”. It attaches a new container to a pod that is already running, without restarting it:

kubectl debug -it mypod --image=busybox:1.36 --target=app

What happens under the hood: kubectl sends a PATCH to the pod’s ephemeralcontainers subresource, the kubelet starts the new container inside the same pod sandbox, and because of -it kubectl attaches your terminal to it. The ephemeral container automatically shares the pod’s network namespace, so localhost, the pod IP and the pod’s DNS configuration are exactly what the application sees. That alone makes it the fastest way to test connectivity from a pod’s point of view.

The –target flag and the process namespace

By default, containers in a pod do not see each other’s processes. The --target=<container> flag tells the runtime to put the debug container into the process namespace of the named container. With it, ps inside the debug container shows the application’s processes, and you can inspect them:

/ # ps aux
PID   USER     TIME  COMMAND
    1 65532     0:12 /app/server --port=8080
   23 root      0:00 sh
   29 root      0:00 ps aux

Two caveats from the official documentation. First, --target must be supported by the container runtime; containerd and CRI-O support it, but if it is not supported the ephemeral container may fail to start or start with an isolated process namespace, and ps will only show its own processes. Second, --target is only necessary when the pod does not already set shareProcessNamespace: true. If it does, every container, including the ephemeral one, already sees all processes.

Reading the application’s filesystem through /proc

Once you share the target’s process namespace, the application’s root filesystem is reachable through the proc filesystem, even though the debug container has its own image:

/ # ls /proc/1/root/app
config.yaml  server
/ # cat /proc/1/root/app/config.yaml

This is the trick that makes distroless images debuggable: you get your tools from busybox and the files from the target. If you get Permission denied on /proc/1/root, the debug container lacks the ptrace access it needs to look into another user’s process. Use --profile=general (which adds SYS_PTRACE) or, if you really need it, --profile=sysadmin. The profiles are covered in detail below.

Naming, detaching and reattaching

If you do not pass -c, kubectl generates a name like debugger-8xzrl. Name it yourself so you can find it again:

kubectl debug mypod -c debugger --image=nicolaka/netshoot --target=app -it

If your terminal disconnects, the container keeps running while its main process is alive. You can reattach with:

kubectl attach mypod -c debugger -it

When you type exit, the shell (the container’s main process) ends and the ephemeral container moves to Terminated. It is never restarted, and a terminated ephemeral container cannot be started again; you add a new one with a different name instead. Running kubectl describe pod mypod shows every ephemeral container ever added under an Ephemeral Containers: section.

Mounting volumes in a debug container

A common follow-up question is how to make kubectl debug mount a volume. The answer has two parts. You cannot add a new volume, because the pod spec’s volume list is immutable after creation. You can, however, mount a volume that the pod already declares, through a custom profile (see the profiles section) that sets volumeMounts:

# debug-mounts.yaml
volumeMounts:
- name: data
  mountPath: /data

kubectl debug mypod -it --image=busybox:1.36 --target=app \
  --profile=general --custom=debug-mounts.yaml

The name must match an entry in the pod’s spec.volumes. subPath mounts are not allowed for ephemeral containers. If what you need is just to read files the application sees, /proc/1/root is usually simpler than a mount.

Mode 2: Debugging a Copy of the Pod

Ephemeral containers are useless when the application container is crash-looping: there is no stable process to target, and you cannot change the application’s command. For those cases kubectl debug can create a copy of the pod with modifications, using --copy-to.

Changing the command of a crashing container

This is the standard fix for CrashLoopBackOff investigation. Create a copy where the failing container runs a shell instead of its entrypoint:

kubectl debug mypod -it --copy-to=mypod-debug --container=app -- sh

You now have a shell in a container with the same image, environment variables, volumes, and service account as the crashing one. From there you can inspect config files, check that mounted secrets exist, and run the original entrypoint by hand to watch it fail with full output.

The --container flag matters. The documentation is explicit: to change the command of an existing container you must name it, otherwise kubectl debug creates a new container to run the command you specified. This only works if the image has a shell; for distroless images, combine it with --set-image (next section) to swap in a debug variant.

Swapping images

--set-image changes images in the copy using the same container=image syntax as kubectl set image:

# Replace one container's image with a debug build
kubectl debug mypod -it --copy-to=mypod-debug --set-image=app=myregistry/app:1.4.2-debug

# Replace every container's image
kubectl debug mypod --copy-to=mypod-debug --set-image=*=ubuntu

Swapping to a :debug tag of the same application (many distroless-based images publish one with a busybox shell) is often the cleanest way to get a shell into an otherwise shell-less image.

Adding a debug container to the copy

You can also add a brand-new container to the copy, which works even when your cluster blocks ephemeral containers:

kubectl debug mypod -it --image=ubuntu --copy-to=mypod-debug --share-processes

--share-processes enables shareProcessNamespace in the copy so the debug container can see the application’s processes. It defaults to true when you use --copy-to, so you only need to spell it out for readability.

What gets copied and what does not

The copy is a standalone pod with no owner reference: it is not part of the original Deployment or ReplicaSet, so the controller does not manage it and it will not be replaced if it dies. Several fields are intentionally stripped so the copy behaves as a lab specimen and not as a production replica:

Two scheduling flags are worth knowing. --same-node schedules the copy on the same node as the original, which matters when you suspect a node-specific problem. --replace deletes the original pod after creating the copy; use it with care, since for a Deployment the controller will simply create a new replica.

Remember that the copy is a real pod consuming real resources. Delete it when you are done:

kubectl delete pod mypod-debug

Mode 3: Debugging a Node with kubectl debug node

When the problem is below the pod (kubelet errors, full disks, container runtime state, host networking), you can get a shell on a node without SSH:

kubectl debug node/worker-3 -it --image=ubuntu

kubectl creates a pod named like node-debugger-worker-3-pdx84 pinned to that node. According to the documentation, the container runs in the host’s IPC, network, and PID namespaces, and the node’s root filesystem is mounted at /host. That gives you access to host files such as /host/var/log and to host processes via ps.

The pod is not privileged by default, so reading some process information may fail and chroot /host may not work. If you need full access, use the sysadmin profile:

kubectl debug node/worker-3 -it --image=ubuntu --profile=sysadmin
# inside the container
chroot /host
crictl ps
journalctl -u kubelet --since "10 min ago"

This is particularly useful on immutable operating systems. On Talos Linux, for example, there is no SSH daemon at all, so node-level inspection goes through talosctl or a node debug pod; our Talos Linux guide covers that model.

Node debug pods are not cleaned up automatically. Delete them explicitly:

kubectl delete pod node-debugger-worker-3-pdx84

Debugging Profiles: –profile Explained

Every kubectl debug session applies a profile that decides the security context of the debug container. The profile matters because it determines what you can do (read other processes, capture packets, chroot to the host) and whether Pod Security admission accepts your debug container at all.

The descriptions follow the design in KEP-1441, which defines the profiles. The practical mapping is simple:

• Use general for everyday debugging; it is enough to read /proc/1/root and attach strace or a debugger to the application.
• Use netadmin with nicolaka/netshoot when you need tcpdump, iptables inspection or raw sockets.
• Use baseline or restricted when the namespace enforces those Pod Security Standards; other profiles will be rejected there.
• Use sysadmin only for node work or when nothing else works, and treat it like handing out root.

The default profile changed

For years, running kubectl debug without --profile meant legacy, and recent versions printed a warning about it. Following KEP-1441, general is the default from kubectl 1.36, and the legacy profile is scheduled for removal in 1.39. Note that the behavior depends on your kubectl client version, not on the cluster version. If you have scripts that rely on the old behavior, set --profile explicitly; it is good practice anyway because the profile is part of what you are asking the cluster to allow.

Custom profiles with –custom

Since Kubernetes 1.32, custom profiles are stable. A custom profile is a partial container spec in YAML or JSON that is merged on top of the static profile:

# custom-profile.yaml
env:
- name: JAVA_TOOL_OPTIONS
  value: "-XX:+UnlockDiagnosticVMOptions"
securityContext:
  capabilities:
    add:
    - NET_ADMIN
    - SYS_TIME

kubectl debug -it mypod --image=busybox:1.36 --target=app \
  --profile=general --custom=custom-profile.yaml

The documented constraints: a custom profile can only modify the container spec, not the pod spec, and it cannot change the container’s name, image, command, lifecycle or volumeDevices. Everything else, including env, securityContext, volumeMounts and workingDir, is fair game, which is how the volume mount example earlier works.

RBAC: What Permissions kubectl debug Needs

Most “kubectl debug does not work” reports are RBAC problems. The permissions differ by mode:

A minimal namespaced Role for ephemeral debugging looks like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ephemeral-debugger
  namespace: payments
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["patch", "update"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create"]

The error pods "mypod" is forbidden: User "dev" cannot patch resource "pods/ephemeralcontainers" means exactly the second rule is missing.

Be deliberate about who gets this. Patching pods/ephemeralcontainers lets a user run an arbitrary image inside any pod in the namespace, sharing its network and, with --target, its process space and filesystem. That is effectively the same trust level as pods/exec, and it should be granted and audited the same way. The Kubernetes security best practices article covers how to scope this kind of access, and namespace isolation is what keeps it contained.

Pod Security Admission and kubectl debug

Pod Security admission evaluates ephemeral containers too. In a namespace labeled pod-security.kubernetes.io/enforce=restricted, a debug container that runs as root or adds capabilities is rejected, and the error mentions the violated rule, for example allowPrivilegeEscalation != false or unrestricted capabilities. The fix is to pick the matching profile:

kubectl debug -it mypod --image=busybox:1.36 --target=app --profile=restricted

With restricted, the debug container runs as non-root with all capabilities dropped, so you lose SYS_PTRACE and most of /proc/1/root becomes unreadable unless the application runs as the same UID. That is a genuine limitation of debugging under the restricted standard, not a kubectl bug.

Node debugging has the same issue in a different place: the node debug pod uses host namespaces and a hostPath mount, which the baseline and restricted standards forbid. Since kubectl creates it in your current namespace, run node debugging from a namespace that allows it (typically a dedicated, privileged-labeled namespace such as node-debug) with -n.

Limitations You Should Know Before Using It in Production

You cannot remove an ephemeral container. Like regular containers, an ephemeral container cannot be changed or removed once added. There is no kubectl delete for it. When its process exits it stays in the pod’s status as Terminated until the pod itself is deleted. If a stack of terminated debug containers bothers you, the only cleanup is replacing the pod, for example with kubectl rollout restart deployment/<name>.

No resources, no ports, no probes. Ephemeral containers cannot declare resources, ports, or probes, because pod resource allocations and networking are fixed at creation. The debug container runs without its own requests, so a heavy tool (a JVM heap dump, a large tcpdump capture written to disk) competes with the application for the pod’s resources and node capacity. Keep sessions short and output small, and read resource requests and limits if you are unsure how much headroom the pod has.

No kubectl edit. Ephemeral containers are added through a dedicated API subresource, not by editing pod.spec, so you cannot add one with kubectl edit or kubectl apply.

Static pods are not supported. Ephemeral containers do not work on static pods, which includes control-plane components on kubeadm clusters. Use node debugging for those.

It only targets pods and nodes. kubectl debug accepts a pod, a node/<name>, or a file with -f. To debug a Deployment, pick one of its pods first, for example with kubectl get pods -l app=myapp.

The copy is not the original. A --copy-to pod runs on a fresh sandbox, has no live traffic, and may land on a different node. It reproduces startup and configuration problems very well and runtime or traffic-dependent problems poorly.

Choosing a Debug Image

The debug container can use any image your nodes can pull, so pick based on the job. Pin tags so your team gets the same toolset every time.

Practical Recipes

Debugging a CrashLoopBackOff pod

Start with the logs of the previous run, since they often contain the answer:

kubectl logs mypod -c app --previous

If the logs are empty or unhelpful, create a copy that does not start the app, then run it by hand:

kubectl debug mypod -it --copy-to=mypod-debug --container=app -- sh
/ # env | sort
/ # ls -la /etc/config
/ # /app/server --port=8080

For shell-less images, swap to a debug variant with --set-image, or add a busybox container with --target and read the files through /proc/1/root, the pattern covered step by step in debugging distroless containers.

Checking network connectivity from a pod

The ephemeral container shares the pod’s network namespace, so tests run from the pod’s real IP with its real DNS configuration and NetworkPolicies:

kubectl debug -it mypod --image=nicolaka/netshoot --profile=netadmin
~ # dig +search my-service
~ # curl -sv http://my-service.payments.svc:8080/healthz
~ # tcpdump -i any -nn port 5432

If the application sits behind a service mesh sidecar, remember that outbound traffic goes through the proxy; the article on Istio proxy DNS explains why DNS results can differ between the app and a plain debug container.

kubectl debug vs kubectl exec vs Pod Copy

The rule of thumb: use exec if the container is running and has what you need; use an ephemeral container when you need tools the image lacks while observing the live process; use a copy when the container cannot start or you need to change how it starts; use node debugging when the problem is not inside the pod. If you prefer a GUI, desktop clients like Freelens wrap exec and logs, but the three kubectl debug modes remain the most capable option.

Frequently Asked Questions

Conclusion

kubectl debug is three tools behind one command. The ephemeral container mode is what you reach for when a running pod misbehaves and its image has nothing useful in it; the copy mode is for containers that cannot start or need a different command or image; and the node mode replaces SSH for anything below the pod. Set --profile explicitly, grant pods/ephemeralcontainers as carefully as pods/exec, and remember that anything you attach to a live pod stays in its status until the pod is replaced. With those habits in place, a shell-less image stops being an obstacle to debugging.

If your Kubernetes HPA scales down slowly, it is almost certainly doing exactly what it was designed to do. By default the HorizontalPodAutoscaler waits for a 300-second stabilization window before removing pods: it keeps the highest replica recommendation it has computed in the last five minutes, and only drops below that once every recommendation in the window agrees. Add the 15-second sync period, the metrics-server scrape interval and a 10% tolerance band, and “traffic stopped at 10:00, pods started going away at 10:06” is the normal, healthy outcome.

The fix, if you want a different outcome, is the behavior field in autoscaling/v2. This is the short version for “scale down faster”:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60   # default is 300
      policies:
        - type: Percent
          value: 50
          periodSeconds: 30

The rest of this article explains every moving part of HPA scale-down behavior — so you can tell the difference between an HPA that is slow on purpose, one that is stuck, and one that is flapping — and gives tested recipes for each case. If your HPA runs on memory, the memory-specific details (why memory does not fall after load drops, requests vs limits) are covered in Kubernetes HPA on memory; this page is about the scale-down machinery itself, which is the same for every metric.

What Happens Between “Load Dropped” and “Pod Removed”

When people say the HPA is slow to scale down, they usually mean the total elapsed time from the moment load falls to the moment a pod is terminated. That time is the sum of several independent delays, and it helps to see them in order:

1. Metric collection. metrics-server scrapes the kubelets every 15 seconds by default, and the kubelet’s CPU figure is itself a rate over a short window. A drop in load shows up in the resource metrics API somewhere between a few seconds and ~30 seconds later.
2. HPA sync period. The HPA controller in kube-controller-manager evaluates every HPA every 15 seconds (--horizontal-pod-autoscaler-sync-period). Add up to another 15 seconds.
3. Tolerance. If the ratio between the current metric and the target is within 10% of 1.0, the controller does nothing. A gradual decline can spend minutes inside that band.
4. Stabilization window. The controller records a recommendation every sync and, for scale-down, applies the highest recommendation from the last 300 seconds. This is the big one: five minutes minimum after the last high recommendation.
5. Scaling policies. Once the window allows it, the default scale-down policy removes up to 100% of the excess pods every 15 seconds — so this step is fast by default, unless you slowed it down.
6. Pod termination. The Deployment controller deletes pods, which then go through preStop hooks and terminationGracePeriodSeconds. A 60-second graceful shutdown is 60 more seconds before the pod is actually gone.

So with defaults, a clean drop from 10 replicas to 3 typically completes between 5.5 and 6.5 minutes after load falls. If you are seeing 20 minutes, or never, something else is going on — keep reading.

The Default HPA Behavior (and Why Scale-Down Is Deliberately Slow)

When you omit behavior, the HPA behaves as if you had written this (the values are straight from the Kubernetes documentation):

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
      - type: Percent
        value: 100
        periodSeconds: 15
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15
    selectPolicy: Max

The asymmetry is intentional. Scaling up late costs you latency and errors; scaling down late costs you a few minutes of spare capacity. So scale-up reacts immediately (no window, double the fleet or add four pods every 15 seconds, whichever is larger) and scale-down waits five minutes to make sure the drop is real.

The cluster-wide default for the scale-down window comes from the --horizontal-pod-autoscaler-downscale-stabilization flag on kube-controller-manager (5 minutes). On managed platforms — EKS, GKE, AKS — you cannot change controller-manager flags, which is exactly why the per-HPA behavior field exists. Always tune in the HPA object, never rely on the cluster flag.

How stabilizationWindowSeconds Actually Works

The stabilization window is the most misunderstood field in the HPA spec, because the name suggests a delay or a cooldown. It is neither. It is a rolling maximum (for scale-down) or rolling minimum (for scale-up) over the replica recommendations computed during the window.

Concretely: every 15 seconds the controller computes a desired replica count from the metrics and stores it with a timestamp. For scale-down, before acting it looks at all recommendations from the last stabilizationWindowSeconds and uses the highest one. A single high recommendation four minutes ago is enough to hold the fleet at that size for one more minute.

Walk through an example with the default 300-second window:

This is why the scale-down happens in steps that mirror the recommendations from five minutes earlier, and why a brief spike during the quiet period resets the clock. While the window is holding replicas up, kubectl describe hpa shows the condition AbleToScale True ScaleDownStabilized with the message “recent recommendations were higher than current one, applying the highest recent recommendation”. If you see that message, the HPA is not broken — it is waiting.

A few properties of the field worth knowing:

• The value range is 0 to 3600 seconds. Anything above one hour is rejected by API validation.
• stabilizationWindowSeconds: 0 on scale-down means “act on the current recommendation immediately”. Useful for batch workers; dangerous for anything with bursty traffic.
• The same field exists under scaleUp, where it works as a rolling minimum. Setting a scale-up window of 60 seconds means the HPA only scales up if the last minute of recommendations all agree — a good way to ignore one-sample spikes, at the cost of reacting a minute later.
• The window is held in the controller’s memory. When kube-controller-manager restarts or fails over, the recommendation history is lost, and the next scale-down can happen sooner than you expect.

Scaling Policies: Pods, Percent, periodSeconds and selectPolicy

Policies limit how fast the replica count can change once the stabilization window has allowed a change. Each policy says “in any periodSeconds window, change by at most value pods (or value percent of the pods)”.

type: Pods is an absolute number: value: 2 with periodSeconds: 60 means at most two pods removed per minute, regardless of fleet size. This is the predictable option and the one I default to for scale-down.

type: Percent is relative to the replica count at the start of the period: value: 25 with periodSeconds: 60 on a 40-pod fleet allows removing 10 pods in that minute, but on a 4-pod fleet only 1. Percent policies scale with the fleet, which is what you want for large deployments and a nuisance for small ones.

periodSeconds can go up to 1800 (30 minutes). The controller tracks actual scaling events within the period, so a policy of 1 pod per 300 seconds really means “no more than one removal in any rolling five-minute span”, even across multiple HPA syncs.

When you list several policies, selectPolicy decides which one wins:

• Max (the default) picks the policy that allows the biggest change. With Pods: 4 and Percent: 100, a 2-pod fleet can grow by 4 and a 20-pod fleet by 20.
• Min picks the policy that allows the smallest change. This is the conservative choice for scale-down: “at most 10% or 2 pods per minute, whichever is smaller”.
• Disabled turns scaling off in that direction completely. scaleDown.selectPolicy: Disabled means the HPA will only ever add pods.

When a policy is what’s holding the fleet back, kubectl describe hpa shows ScalingLimited True ScaleDownLimit — “the desired replica count is decreasing faster than the maximum scale rate”. That is different from ScaleDownStabilized, and it tells you which knob to turn: the window or the policy.

Tolerance and Rounding: Why the HPA Sometimes Never Scales Down

A frequent complaint is not “the HPA is slow”, but “the HPA never scales down, even though usage is clearly below target”. Two pieces of arithmetic cause most of these cases.

The 10% tolerance band. The controller computes ratio = currentMetricValue / targetValue and skips any action if the ratio is within ±0.1 of 1.0. With a 70% CPU target, an average of 64% gives a ratio of 0.914 — inside the band, so nothing happens. Your fleet can sit at 64% indefinitely, 6 points below target, without a single pod removed.

Ceiling rounding at small replica counts. The desired count is ceil(currentReplicas × ratio). Rounding up is safe for scale-up, but it makes scale-down surprisingly sticky on small fleets. With 3 replicas, a 70% target and an average of 50%:

desired = ceil(3 × 50 / 70) = ceil(2.14) = 3

The ratio (0.71) is well outside the tolerance, yet the result is still 3. To go from 3 to 2 pods, the average must drop to 46.7% or below (2/3 × 70), because those 2 pods would then run at 70%. The HPA is refusing to remove a pod that would push the remaining ones over target — which is correct, but it looks like a bug when you are staring at a dashboard that says 50%. The smaller the fleet, the bigger this effect: going from 2 to 1 requires the average to fall to half the target.

Configurable tolerance per HPA. Until recently the 10% was a cluster-wide constant (--horizontal-pod-autoscaler-tolerance), which on managed clusters meant you could not change it at all. The HPAConfigurableTolerance feature adds a tolerance field per direction:

behavior:
  scaleUp:
    tolerance: 0.05    # react when 5% above target
  scaleDown:
    tolerance: 0.15    # only scale down when 15% below target

The feature went alpha in Kubernetes 1.33, beta in 1.35 (still disabled by default in beta, so managed providers generally did not expose it) and GA in 1.37, where it is always on. On a cluster older than 1.37, check the feature gate before relying on it — an API server that does not know the field silently drops it. A lower scale-up tolerance with a higher scale-down tolerance is a sensible pairing for latency-sensitive services: react early to growth, ignore small declines.

Why an HPA Stops Scaling Down: The Checklist

When the HPA is not scaling down at all — not slowly, not at all — work through these in order. They cover the cases I have actually seen in production.

minReplicas is reached. Obvious, but check it first. ScalingLimited True TooFewReplicas in the conditions confirms it.

Another metric is holding it up. With multiple metrics, the HPA computes a desired count for each and uses the largest. CPU can be at 10% while memory or a queue-length metric keeps the fleet at its current size. kubectl describe hpa lists each metric’s current value; look for the one still near its target. For memory specifically, the reasons it does not fall after load drops are covered in the HPA memory guide.

One metric is failing. This one is subtle and documented: if any metric cannot be fetched and the remaining metrics suggest a scale-down, the controller skips scaling entirely. Scale-up still works in that situation; scale-down does not. A broken Prometheus Adapter query or a missing custom metric can therefore freeze your fleet at its peak size for days. Look for FailedGetPodsMetric, FailedGetExternalMetric or FailedGetResourceMetric events.

Pods with missing metrics. When some pods have no metrics yet (new pods, or pods on a node where metrics-server is failing), the controller recomputes conservatively and assumes those pods are at 100% of target when evaluating a scale-down. A handful of pods with no metrics can keep the average above the scale-down threshold.

Rounding on a small fleet. See the previous section. On 2 or 3 replicas, the average may need to fall far below the target before a pod is removed.

Scale-down is disabled. Someone may have set scaleDown.selectPolicy: Disabled, or a very long window. kubectl get hpa <name> -o yaml and read the behavior block — including defaults, which the API server fills in.

Something else owns the replica count. A GitOps tool that sets spec.replicas on the Deployment, a KEDA ScaledObject targeting the same Deployment, or a second HPA will fight the HPA. ArgoCD will even show the Deployment as out of sync every time the HPA changes it. Remove replicas from the manifest you apply, and make sure exactly one autoscaler targets each workload.

Requests are wrong. Utilization is usage divided by request. If requests are set far below real idle usage, “idle” is already above target and the HPA will never scale down. The fix is to right-size requests, not to tune the HPA — see Kubernetes resource requests and limits.

Why an HPA Scales Down Too Fast (or Flaps)

The opposite problem shows up on bursty workloads: the HPA removes pods, traffic comes back two minutes later, the HPA adds them again, and new pods take a while to be useful. Every cycle costs cold starts, connection re-balancing and, often, a latency spike.

Common causes:

• A short or zero scale-down window. Teams set stabilizationWindowSeconds: 0 to “fix” slowness and create flapping. If traffic has a periodicity shorter than your window — a cron that fires every 10 minutes, a batch that arrives every quarter hour — the window must be longer than that period.
• Aggressive Percent policies. Percent: 100 allows removing all excess pods at once. A Pods policy that removes one or two per period turns a cliff into a staircase, which gives you time to notice a returning load.
• Slow-starting pods. If a new pod takes three minutes to warm up — JVM JIT compilation, cache loading, connection pools — every scale-down you later have to reverse costs three minutes of under-capacity. The HPA has two cluster-level safeguards for CPU (--horizontal-pod-autoscaler-cpu-initialization-period, 5 minutes, and --horizontal-pod-autoscaler-initial-readiness-delay, 30 seconds) that ignore CPU samples from pods that are still starting. They only help if your readiness probe does not report Ready before the warm-up is done. Use a startupProbe that holds the pod back until it can actually serve.

HPA Behavior Recipes (autoscaling/v2)

All of these are complete manifests you can apply as-is after changing the target and metrics. They use CPU for brevity; the behavior block is the part that matters, and it works identically with memory, custom or external metrics.

Scale down faster

For stateless services with fast startup and load that drops cleanly — internal APIs, dev and staging environments, anything where idle capacity costs more than an occasional cold start:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-fast-down
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 50
          periodSeconds: 30
      selectPolicy: Max

The fleet can shrink one minute after load drops, halving at most every 30 seconds. Going below 60 seconds rarely buys anything real: metric lag already eats 15-30 seconds of it.

Scale down gradually (fast up, slow down)

The safe default for customer-facing services, bursty traffic and slow-starting runtimes:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-gradual
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
        - type: Pods
          value: 4
          periodSeconds: 30
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Pods
          value: 2
          periodSeconds: 120
        - type: Percent
          value: 10
          periodSeconds: 120
      selectPolicy: Min

Scale-up doubles the fleet (or adds 4, whichever is more) every 30 seconds. Scale-down waits until ten minutes of recommendations agree, then removes at most 2 pods or 10% every two minutes, whichever is smaller. A drop from 50 to 10 pods takes about 40 minutes, which is exactly the point: if traffic comes back, you are still mostly provisioned.

Never scale down automatically

For workloads where the HPA is there to absorb peaks, and scale-down should happen only through a deploy or a human decision — stateful consumers that rebalance partitions on every change, services with very expensive warm-up, or the first weeks of a new HPA you do not yet trust:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: consumer-no-down
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: consumer
  minReplicas: 4
  maxReplicas: 24
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      selectPolicy: Disabled

The cost: the fleet sits at its high-water mark until someone lowers maxReplicas or redeploys.

KEDA: cooldownPeriod Is Not What Controls Scale-Down

If you use KEDA, you have probably tried cooldownPeriod to slow down or speed up scale-down and seen no effect. That is because cooldownPeriod (default 300 seconds) only applies to the transition from 1 replica to 0: how long KEDA waits after the last active trigger before scaling the workload to zero. Every change between N and 1 replicas is made by the HPA that KEDA creates, and is controlled by the same behavior field described above — passed through the ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker
spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 0
  maxReplicaCount: 40
  cooldownPeriod: 300          # only 1 -> 0
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:                # everything between maxReplicaCount and 1
        scaleDown:
          stabilizationWindowSeconds: 120
          policies:
            - type: Percent
              value: 25
              periodSeconds: 60
  triggers:
    - type: rabbitmq
      metadata:
        queueName: jobs
        mode: QueueLength
        value: "20"
      authenticationRef:
        name: rabbitmq-auth

For queue-driven workers, a short window is usually fine: queue length is a leading signal, not a lagging one like CPU. More on KEDA triggers in event-driven autoscaling with KEDA.

How to Diagnose HPA Scale-Down with kubectl

Three commands tell you almost everything.

Watch the HPA live to see the relationship between the metric and the replica count over time:

kubectl get hpa api -w

Describe it to see the conditions and the recent events — this is where the “why” lives:

kubectl describe hpa api

A healthy HPA waiting on its stabilization window looks like this:

Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  18% (90m) / 70%
Min replicas:                                          2
Max replicas:                                          30
Behavior:
  Scale Up:
    Stabilization Window: 0 seconds
    Select Policy: Max
    Policies:
      - Type: Pods     Value: 4    Period: 15 seconds
      - Type: Percent  Value: 100  Period: 15 seconds
  Scale Down:
    Stabilization Window: 300 seconds
    Select Policy: Max
    Policies:
      - Type: Percent  Value: 100  Period: 15 seconds
Deployment pods:       10 current / 10 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range

How to read the conditions:

If ScalingActive is False, stop tuning behavior: the HPA is not receiving metrics, and the events at the bottom of the describe output (FailedGetResourceMetric, FailedComputeMetricsReplicas) say why.

Check the raw metrics the HPA sees, to rule out metrics-server lag or missing pods:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods" | jq '.items[] | {name: .metadata.name, cpu: .containers[].usage.cpu}'
kubectl top pods -l app=api

If some pods are missing from that output, you have found your “missing metrics treated as 100%” problem.

Pod Scale-Down Is Not Node Scale-Down

One last source of confusion: even when the HPA removes pods promptly, the nodes do not disappear at the same time, so the cloud bill does not drop when you expect. Node removal is a separate decision made by Cluster Autoscaler or Karpenter, with its own delays (Cluster Autoscaler waits 10 minutes by default before removing an underutilized node) and its own blockers, such as PodDisruptionBudgets and pods that cannot be moved. If cost is the reason you want faster scale-down, tune both layers — the comparison of how each one consolidates nodes is in Cluster Autoscaler vs Karpenter.

And before tuning behavior at all, make sure the metric is the right one: a CPU target on a service that is really bound by I/O or a downstream dependency will never scale down cleanly, no matter what the window is. The broader design questions — which metric, which target, when HPA is the wrong tool — are in Kubernetes HPA best practices.

Frequently Asked Questions

Conclusion

A Kubernetes HPA that scales down slowly is, nine times out of ten, the default 300-second stabilization window doing its job, plus metric and sync lag on top. Treat that as a starting point, not a bug: shorten the window for services that start fast and cost money when idle, lengthen it and add a Pods policy for services where every cold start hurts, and disable scale-down entirely where only a human should make that call.

When the HPA does not scale down at all, stop tuning behavior and read kubectl describe hpa. The conditions tell you whether you are looking at stabilization (ScaleDownStabilized), a rate limit (ScaleDownLimit), a floor (TooFewReplicas) or broken metrics (ScalingActive False) — and the tolerance and rounding arithmetic explains the rest. Get that right, and the HPA becomes predictable enough that you can reason about its next move instead of waiting for it.

Every week brings another “we built a Kubernetes-native X” announcement. Some of those projects will be running your production clusters in three years; most will be archived repositories with a “no longer maintained” banner. The CNCF maturity level — Sandbox, Incubating, Graduated — is the most widely used shortcut for telling the two apart, and it is also one of the most misread.

The labels are not marketing tiers. Each one corresponds to a documented set of criteria that the Technical Oversight Committee (TOC) checks in a due-diligence process, and each one is a floor, not a guarantee. This article goes through what the TOC actually requires at each level as of 2026, what the levels do not tell you, the 2025–2026 events that show the ladder moving in both directions, and a practical framework for turning a badge into an adoption decision.

The Short Answer

Sandbox means the TOC judged the idea worth hosting; it says almost nothing about stability, security posture or whether anyone will maintain it next year. Incubating means the project has documented governance, at least three independent production adopters interviewed by the TOC, a security self-assessment and a passing OpenSSF badge — it is the level at which the CNCF starts actively evaluating adoption. Graduated adds maintainers from at least two organisations, an org-balance mechanism in governance and a completed third-party security audit — it is the level at which the foundation is willing to say the project has demonstrated production readiness at scale.

If you need one rule: Sandbox is where you experiment, Incubating is where you can build with a contingency plan, Graduated is where you standardise.

What the TOC Actually Requires

The criteria live in the TOC repository, and since 2024 the detailed lists sit inside the incubation and graduation application templates rather than a single criteria document. These are the items that matter for an adopter, quoted from the current templates.

Three things in that table are worth reading twice.

First, Sandbox has almost no technical bar. The published reasons applications get closed without review are administrative: not Apache 2.0, no MAINTAINERS file with employer affiliations, being a reference architecture rather than a reusable project, or being a subproject without the parent’s consent to split. Organisation diversity is explicitly “not a requirement, but considered during review”. The TOC reviews applications in a non-public session roughly every two months, working through seven to ten at a time in first-in-first-out order. There is no security review, no adopter interview and no architecture review at this stage — TAG-led General Technical Reviews are optional and “do not result in a label or a required action for applicants”.

Second, the OpenSSF badge requirement is “passing” at both Incubating and Graduated, not silver or gold. If a project advertises a gold badge, that is the project going beyond the bar, not the CNCF demanding it.

Third, the third-party security audit only arrives at Graduation. An Incubating project has assessed itself. That is not nothing — the self-assessment template is thorough and TAG Security reviews it — but it is not an external audit of the code you are about to run with cluster-admin privileges.

What the Levels Do Not Mean

Sandbox is not an endorsement

The lifecycle document describes Sandbox projects as ones where “failure is a possibility” and that “are expected to undergo significant changes, including breaking changes to their functionality”. It adds that they “might still be used in production by a few organizations on a case to case basis” — which is the TOC’s careful way of saying that it is possible, and that the risk is entirely yours.

The practical consequence: a Sandbox badge on a README tells you the licence is clean and the maintainers are identifiable. It does not tell you the API will exist in the same shape next quarter.

Incubating is not “almost graduated”

Projects can sit at Incubating for a very long time. Thanos entered the CNCF in July 2019 and moved to Incubating on 19 August 2020; six years later it is still Incubating and still perfectly healthy. Contrast KEDA — Sandbox March 2020, Incubating August 2021, Graduated August 2023 — or Kyverno, which took from November 2020 to March 2026 to make the full journey. Time at a level is a function of governance work and audit scheduling as much as technical maturity, so “still Incubating” is not a warning sign on its own.

Graduated is not “right for you”

Graduation says the project has cleared the foundation’s bar for governance, security and adoption breadth. It does not say the project fits your scale, your team or your problem. Cilium, Argo, Kyverno and KEDA are all Graduated; you should still not run all four if you have one cluster and three engineers. The badge removes the “will this be abandoned” question from your evaluation; it does not answer any of the others.

And some things you assume are CNCF projects are not

Karpenter is a good example of the trap. It is hosted under kubernetes-sigs as a subproject of SIG Autoscaling, which means it is governed by the Kubernetes project, not by the CNCF TOC as a standalone project with its own maturity level. It has no Sandbox, Incubating or Graduated badge to check, and any blog post that assigns it one is guessing. When you evaluate it, you are evaluating a Kubernetes SIG subproject, which is a different (and generally stricter) governance regime. The Cluster Autoscaler vs Karpenter comparison covers the technical side.

The Ladder Moves in Both Directions

2026 has been an unusually busy year for graduations: Dragonfly (January), Kyverno (16 March, announced 24 March), OpenTelemetry (May), Cloud Native Buildpacks (August), Kubeflow (August) and Karmada (8 September). Karmada is the clean illustration of the full path — Sandbox September 2021, Incubating December 2023, Graduated September 2026, with the announcement explicitly noting that the project “completed a third-party security audit, established a formal steering committee to ensure transparent governance” to get there.

The archive side is less publicised and more instructive. In 2025 the TOC archived two batches of Sandbox projects: Nocalhost, SuperEdge and KubeDL in March, then Merbridge, Sealer, Teller, DevStream and OpenELB in May. All eight were Sandbox projects, several of them with a single corporate sponsor, and each went through the documented process — a project-health issue, a two-week final comment period, then a two-week TOC vote requiring a 2/3 majority.

Archiving is also not necessarily the end. OpenEBS was archived after a TOC vote in early 2024 and re-admitted to Sandbox in October 2024 with a rebuilt maintainer team, because “any project can be reactivated into CNCF by following the normal Project Lifecycle & Process”. What an archived project loses is real, though: it “may not be used any more” to advertise its former status, and it disappears from the landscape, DevStats and CLOMonitor.

For an adopter the lesson is simple. Sandbox is the level where the foundation has the least information about the project and the least reason to fight for it. If the project has one employer behind its maintainers, the foundation’s exit process is the one you should be planning for.

Signals You Can Read Yourself

The TOC’s own inputs for health and archiving decisions are public, so you can run the same checks before the TOC does. The archiving policy lists the signals it weighs: the project health dashboard, whether the project still meets current acceptance criteria, community requests, “security responsiveness”, meeting activity, mailing-list engagement and adoption statistics.

Project Health issues. Concerns are filed in cncf/toc with the [HEALTH] prefix. The template describes its purpose as ascertaining “the current activity and health of the project so the TOC may identify the appropriate support and guidance for the project to return to an optimal state of health or determination of archival”. Search the TOC issues for the project name before you adopt; a health issue is a leading indicator by months.

Level-change applications. Incubation and graduation applications are also public issues in cncf/toc. Reading one tells you exactly what the TOC asked, which adopters were interviewed and what was flagged in the security review. This is better due diligence than most vendor RFP responses.

DevStats and LFX Insights. DevStats (devstats.cncf.io) gives contributor and company counts over time; LFX Insights adds organisational affiliation. The metric to watch is not stars but the number of companies contributing in the last year and whether that number is rising.

CLOMonitor. clomonitor.io scores every CNCF project against the same checklist the TOC uses — licence, governance files, security policy, OpenSSF badge, DCO, release signing. A low score at Incubating is worth a question; a low score at Sandbox is normal.

TAG structure. In 2025 the TOC rebooted its Technical Advisory Groups into five: Developer Experience, Infrastructure, Operational Resilience, Security and Compliance, and Workloads Foundation. TAGs no longer perform formal reviews as part of the Sandbox process; that work moved to the TOC’s Project Reviews subproject. If you are reading an old TAG review of a project, check the date — the group that wrote it may no longer exist.

A Practical Evaluation Framework

The maturity level is one input. Here is the checklist that turns it into a decision, with the commands and URLs to answer each question in under an hour.

1. Where does it sit in your stack?

A CNI plugin, a policy engine or a service mesh is foundational: replacing it is a migration project. A dashboard, a CLI plugin or a diagram generator is peripheral: replacing it is an afternoon. The same badge means different things at different layers. Headlamp is a Sandbox project (accepted 17 May 2023) and a perfectly reasonable choice as a cluster UI, because a UI is peripheral — see Headlamp vs FreeLens vs Lens. A Sandbox CNI is a different conversation.

2. Who actually maintains it?

# Committers in the last year, with email domains
git shortlog -sne --since="1 year ago" | head -20

# How concentrated is it? Count distinct email domains among the top 10
git shortlog -sne --since="1 year ago" | head -10 \
  | grep -oE '@[a-z0-9.-]+' | sort | uniq -c | sort -rn

If one domain accounts for more than about 80% of commits, the project’s roadmap is that company’s roadmap, regardless of what the governance document says. This is exactly the “org-balance” question the TOC asks at Graduation; ask it earlier.

3. What does the security posture look like?

# OpenSSF Scorecard: branch protection, signed releases, dependency update tooling, fuzzing
scorecard --repo=github.com/<org>/<project>

Then check for SECURITY.md, whether reported CVEs have advisories with fix versions, and whether release artefacts are signed. For an Incubating project, read the Security Self-Assessment linked from its application issue; for a Graduated one, find the third-party audit report — it is usually published in the project repository or by the auditor.

4. What is the release and compatibility story?

Look at the last twelve months of releases: cadence, whether minor releases carry breaking changes, and whether there is a written deprecation policy. “Fewer breaking changes and more stable, versioned APIs” is literally the TOC’s description of the Incubating transition, so a Sandbox project with a stable API is ahead of its level and an Incubating project without one is behind it.

5. Who else depends on it?

A project that is a dependency of other CNCF projects — or that a managed Kubernetes service ships by default — has a survival guarantee no badge can provide. Cilium is Graduated, but the more important fact for its longevity is that several cloud providers build their networking on it.

6. What is the exit?

Write down, before adopting, what it would take to replace the project. For a policy engine: how many policies, how portable are they (Kyverno’s YAML versus Rego, for example — see enforcing standard and custom policies with Kyverno). For a UI: nothing. For a storage layer: everything. The lower the level, the more this answer matters.

Mapping Level to Adoption Strategy

Start here:
│
├── Is the component foundational (network, storage, policy, identity, mesh)?
│   ├── YES → Graduated by default.
│   │         Incubating only with a documented exit plan and ≥2 maintainer orgs.
│   │         Sandbox: no.
│   └── NO ↓
│
├── Is it core to delivery (CI/CD, GitOps, autoscaling, observability pipeline)?
│   ├── YES → Incubating or Graduated.
│   │         Sandbox only if you are willing to fork or contribute fixes.
│   └── NO ↓
│
├── Is it peripheral (UI, CLI plugin, docs/diagram tooling, developer convenience)?
│   ├── YES → Any level; evaluate on fit and maintainer count, not badge.
│   └── NO ↓
│
└── Not a CNCF project at all (Kubernetes SIG subproject, vendor OSS, no foundation)?
    → Run the same six checks; the absence of a badge is information, not a verdict.

A few corollaries that follow from the criteria rather than from opinion:

• Sandbox projects deserve a re-evaluation date. Put it in the calendar at twelve months. If the project has not applied for Incubation, or has a [HEALTH] issue open, that is when you decide whether to contribute or leave.
• Incubating is the sweet spot for most platform teams. The governance and security documentation exists, adopters have been interviewed, and the project is still moving fast enough to fix the thing you need. Most of what you will evaluate this year sits here.
• Graduated projects earn platform-wide mandates. They are the ones where investment in training, internal documentation and deep expertise has a long-term return, because the foundation has structurally reduced the abandonment risk.
• When a project moves level, re-run the checklist, not the decision. Graduation does not change your architecture. Archiving does — it starts the clock on your exit plan, and the two-week comment period on the archive issue is where you find out whether anyone intends to fork.

Frequently Asked Questions

Conclusion

The CNCF maturity ladder is the best free due diligence in the industry, but only if you read it as what it is: a record of which documented criteria a project has satisfied, checked by a committee that publishes its reasoning. Sandbox tells you the licence is clean. Incubating tells you adopters have been interviewed and governance exists. Graduated tells you an outside party has audited the code and no single company can walk away with it.

None of the three tells you whether the project fits your stack, and none of them replaces the six checks above. Run them, write down the exit, and re-run them when the level changes — in either direction.