Kubernetes

kubectl exec stops working exactly when you need it most: the image has no shell, the container is stuck in CrashLoopBackOff, or the problem lives on the node rather than inside the pod. kubectl debug is the command built for those cases. It can attach a throwaway debug container to a pod that is already running, create a modified copy of a pod you can poke at safely, or drop you into a shell with the node’s filesystem mounted.

This guide covers all three modes, the --profile flag (whose default changed in kubectl 1.36), custom profiles, the RBAC and Pod Security rules that usually get in the way, and the limitations nobody mentions until you hit them, such as the fact that you cannot remove an ephemeral container once it is added. If your specific problem is a distroless image, the deep dive on debugging distroless containers goes further on that case; this article is the general reference for the command itself.

What kubectl debug Does (and Why exec Is Not Enough)

kubectl exec runs a new process inside an existing container. That has three hard requirements: the container must be running, the binary you want (usually sh) must exist in its image, and whatever tools you need must also be in that image. Modern production images break all three. Distroless and scratch images ship no shell, crashing containers are never running long enough to exec into, and nobody wants tcpdump or strace baked into an application image.

kubectl debug removes those requirements by bringing its own image. Depending on the target, it does one of three things:

The command is available in every supported kubectl version. Ephemeral containers, the API feature behind the first mode, have been stable since Kubernetes 1.25, so any cluster you are likely to run today supports them.

Mode 1: Ephemeral Debug Containers in a Running Pod

This is the mode people mean when they search for “kubectl debug container”. It attaches a new container to a pod that is already running, without restarting it:

kubectl debug -it mypod --image=busybox:1.36 --target=app

What happens under the hood: kubectl sends a PATCH to the pod’s ephemeralcontainers subresource, the kubelet starts the new container inside the same pod sandbox, and because of -it kubectl attaches your terminal to it. The ephemeral container automatically shares the pod’s network namespace, so localhost, the pod IP and the pod’s DNS configuration are exactly what the application sees. That alone makes it the fastest way to test connectivity from a pod’s point of view.

The –target flag and the process namespace

By default, containers in a pod do not see each other’s processes. The --target=<container> flag tells the runtime to put the debug container into the process namespace of the named container. With it, ps inside the debug container shows the application’s processes, and you can inspect them:

/ # ps aux
PID   USER     TIME  COMMAND
    1 65532     0:12 /app/server --port=8080
   23 root      0:00 sh
   29 root      0:00 ps aux

Two caveats from the official documentation. First, --target must be supported by the container runtime; containerd and CRI-O support it, but if it is not supported the ephemeral container may fail to start or start with an isolated process namespace, and ps will only show its own processes. Second, --target is only necessary when the pod does not already set shareProcessNamespace: true. If it does, every container, including the ephemeral one, already sees all processes.

Reading the application’s filesystem through /proc

Once you share the target’s process namespace, the application’s root filesystem is reachable through the proc filesystem, even though the debug container has its own image:

/ # ls /proc/1/root/app
config.yaml  server
/ # cat /proc/1/root/app/config.yaml

This is the trick that makes distroless images debuggable: you get your tools from busybox and the files from the target. If you get Permission denied on /proc/1/root, the debug container lacks the ptrace access it needs to look into another user’s process. Use --profile=general (which adds SYS_PTRACE) or, if you really need it, --profile=sysadmin. The profiles are covered in detail below.

Naming, detaching and reattaching

If you do not pass -c, kubectl generates a name like debugger-8xzrl. Name it yourself so you can find it again:

kubectl debug mypod -c debugger --image=nicolaka/netshoot --target=app -it

If your terminal disconnects, the container keeps running while its main process is alive. You can reattach with:

kubectl attach mypod -c debugger -it

When you type exit, the shell (the container’s main process) ends and the ephemeral container moves to Terminated. It is never restarted, and a terminated ephemeral container cannot be started again; you add a new one with a different name instead. Running kubectl describe pod mypod shows every ephemeral container ever added under an Ephemeral Containers: section.

Mounting volumes in a debug container

A common follow-up question is how to make kubectl debug mount a volume. The answer has two parts. You cannot add a new volume, because the pod spec’s volume list is immutable after creation. You can, however, mount a volume that the pod already declares, through a custom profile (see the profiles section) that sets volumeMounts:

# debug-mounts.yaml
volumeMounts:
- name: data
  mountPath: /data

kubectl debug mypod -it --image=busybox:1.36 --target=app \
  --profile=general --custom=debug-mounts.yaml

The name must match an entry in the pod’s spec.volumes. subPath mounts are not allowed for ephemeral containers. If what you need is just to read files the application sees, /proc/1/root is usually simpler than a mount.

Mode 2: Debugging a Copy of the Pod

Ephemeral containers are useless when the application container is crash-looping: there is no stable process to target, and you cannot change the application’s command. For those cases kubectl debug can create a copy of the pod with modifications, using --copy-to.

Changing the command of a crashing container

This is the standard fix for CrashLoopBackOff investigation. Create a copy where the failing container runs a shell instead of its entrypoint:

kubectl debug mypod -it --copy-to=mypod-debug --container=app -- sh

You now have a shell in a container with the same image, environment variables, volumes, and service account as the crashing one. From there you can inspect config files, check that mounted secrets exist, and run the original entrypoint by hand to watch it fail with full output.

The --container flag matters. The documentation is explicit: to change the command of an existing container you must name it, otherwise kubectl debug creates a new container to run the command you specified. This only works if the image has a shell; for distroless images, combine it with --set-image (next section) to swap in a debug variant.

Swapping images

--set-image changes images in the copy using the same container=image syntax as kubectl set image:

# Replace one container's image with a debug build
kubectl debug mypod -it --copy-to=mypod-debug --set-image=app=myregistry/app:1.4.2-debug

# Replace every container's image
kubectl debug mypod --copy-to=mypod-debug --set-image=*=ubuntu

Swapping to a :debug tag of the same application (many distroless-based images publish one with a busybox shell) is often the cleanest way to get a shell into an otherwise shell-less image.

Adding a debug container to the copy

You can also add a brand-new container to the copy, which works even when your cluster blocks ephemeral containers:

kubectl debug mypod -it --image=ubuntu --copy-to=mypod-debug --share-processes

--share-processes enables shareProcessNamespace in the copy so the debug container can see the application’s processes. It defaults to true when you use --copy-to, so you only need to spell it out for readability.

What gets copied and what does not

The copy is a standalone pod with no owner reference: it is not part of the original Deployment or ReplicaSet, so the controller does not manage it and it will not be replaced if it dies. Several fields are intentionally stripped so the copy behaves as a lab specimen and not as a production replica:

Two scheduling flags are worth knowing. --same-node schedules the copy on the same node as the original, which matters when you suspect a node-specific problem. --replace deletes the original pod after creating the copy; use it with care, since for a Deployment the controller will simply create a new replica.

Remember that the copy is a real pod consuming real resources. Delete it when you are done:

kubectl delete pod mypod-debug

Mode 3: Debugging a Node with kubectl debug node

When the problem is below the pod (kubelet errors, full disks, container runtime state, host networking), you can get a shell on a node without SSH:

kubectl debug node/worker-3 -it --image=ubuntu

kubectl creates a pod named like node-debugger-worker-3-pdx84 pinned to that node. According to the documentation, the container runs in the host’s IPC, network, and PID namespaces, and the node’s root filesystem is mounted at /host. That gives you access to host files such as /host/var/log and to host processes via ps.

The pod is not privileged by default, so reading some process information may fail and chroot /host may not work. If you need full access, use the sysadmin profile:

kubectl debug node/worker-3 -it --image=ubuntu --profile=sysadmin
# inside the container
chroot /host
crictl ps
journalctl -u kubelet --since "10 min ago"

This is particularly useful on immutable operating systems. On Talos Linux, for example, there is no SSH daemon at all, so node-level inspection goes through talosctl or a node debug pod; our Talos Linux guide covers that model.

Node debug pods are not cleaned up automatically. Delete them explicitly:

kubectl delete pod node-debugger-worker-3-pdx84

Debugging Profiles: –profile Explained

Every kubectl debug session applies a profile that decides the security context of the debug container. The profile matters because it determines what you can do (read other processes, capture packets, chroot to the host) and whether Pod Security admission accepts your debug container at all.

The descriptions follow the design in KEP-1441, which defines the profiles. The practical mapping is simple:

• Use general for everyday debugging; it is enough to read /proc/1/root and attach strace or a debugger to the application.
• Use netadmin with nicolaka/netshoot when you need tcpdump, iptables inspection or raw sockets.
• Use baseline or restricted when the namespace enforces those Pod Security Standards; other profiles will be rejected there.
• Use sysadmin only for node work or when nothing else works, and treat it like handing out root.

The default profile changed

For years, running kubectl debug without --profile meant legacy, and recent versions printed a warning about it. Following KEP-1441, general is the default from kubectl 1.36, and the legacy profile is scheduled for removal in 1.39. Note that the behavior depends on your kubectl client version, not on the cluster version. If you have scripts that rely on the old behavior, set --profile explicitly; it is good practice anyway because the profile is part of what you are asking the cluster to allow.

Custom profiles with –custom

Since Kubernetes 1.32, custom profiles are stable. A custom profile is a partial container spec in YAML or JSON that is merged on top of the static profile:

# custom-profile.yaml
env:
- name: JAVA_TOOL_OPTIONS
  value: "-XX:+UnlockDiagnosticVMOptions"
securityContext:
  capabilities:
    add:
    - NET_ADMIN
    - SYS_TIME

kubectl debug -it mypod --image=busybox:1.36 --target=app \
  --profile=general --custom=custom-profile.yaml

The documented constraints: a custom profile can only modify the container spec, not the pod spec, and it cannot change the container’s name, image, command, lifecycle or volumeDevices. Everything else, including env, securityContext, volumeMounts and workingDir, is fair game, which is how the volume mount example earlier works.

RBAC: What Permissions kubectl debug Needs

Most “kubectl debug does not work” reports are RBAC problems. The permissions differ by mode:

A minimal namespaced Role for ephemeral debugging looks like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ephemeral-debugger
  namespace: payments
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["patch", "update"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create"]

The error pods "mypod" is forbidden: User "dev" cannot patch resource "pods/ephemeralcontainers" means exactly the second rule is missing.

Be deliberate about who gets this. Patching pods/ephemeralcontainers lets a user run an arbitrary image inside any pod in the namespace, sharing its network and, with --target, its process space and filesystem. That is effectively the same trust level as pods/exec, and it should be granted and audited the same way. The Kubernetes security best practices article covers how to scope this kind of access, and namespace isolation is what keeps it contained.

Pod Security Admission and kubectl debug

Pod Security admission evaluates ephemeral containers too. In a namespace labeled pod-security.kubernetes.io/enforce=restricted, a debug container that runs as root or adds capabilities is rejected, and the error mentions the violated rule, for example allowPrivilegeEscalation != false or unrestricted capabilities. The fix is to pick the matching profile:

kubectl debug -it mypod --image=busybox:1.36 --target=app --profile=restricted

With restricted, the debug container runs as non-root with all capabilities dropped, so you lose SYS_PTRACE and most of /proc/1/root becomes unreadable unless the application runs as the same UID. That is a genuine limitation of debugging under the restricted standard, not a kubectl bug.

Node debugging has the same issue in a different place: the node debug pod uses host namespaces and a hostPath mount, which the baseline and restricted standards forbid. Since kubectl creates it in your current namespace, run node debugging from a namespace that allows it (typically a dedicated, privileged-labeled namespace such as node-debug) with -n.

Limitations You Should Know Before Using It in Production

You cannot remove an ephemeral container. Like regular containers, an ephemeral container cannot be changed or removed once added. There is no kubectl delete for it. When its process exits it stays in the pod’s status as Terminated until the pod itself is deleted. If a stack of terminated debug containers bothers you, the only cleanup is replacing the pod, for example with kubectl rollout restart deployment/<name>.

No resources, no ports, no probes. Ephemeral containers cannot declare resources, ports, or probes, because pod resource allocations and networking are fixed at creation. The debug container runs without its own requests, so a heavy tool (a JVM heap dump, a large tcpdump capture written to disk) competes with the application for the pod’s resources and node capacity. Keep sessions short and output small, and read resource requests and limits if you are unsure how much headroom the pod has.

No kubectl edit. Ephemeral containers are added through a dedicated API subresource, not by editing pod.spec, so you cannot add one with kubectl edit or kubectl apply.

Static pods are not supported. Ephemeral containers do not work on static pods, which includes control-plane components on kubeadm clusters. Use node debugging for those.

It only targets pods and nodes. kubectl debug accepts a pod, a node/<name>, or a file with -f. To debug a Deployment, pick one of its pods first, for example with kubectl get pods -l app=myapp.

The copy is not the original. A --copy-to pod runs on a fresh sandbox, has no live traffic, and may land on a different node. It reproduces startup and configuration problems very well and runtime or traffic-dependent problems poorly.

Choosing a Debug Image

The debug container can use any image your nodes can pull, so pick based on the job. Pin tags so your team gets the same toolset every time.

Practical Recipes

Debugging a CrashLoopBackOff pod

Start with the logs of the previous run, since they often contain the answer:

kubectl logs mypod -c app --previous

If the logs are empty or unhelpful, create a copy that does not start the app, then run it by hand:

kubectl debug mypod -it --copy-to=mypod-debug --container=app -- sh
/ # env | sort
/ # ls -la /etc/config
/ # /app/server --port=8080

For shell-less images, swap to a debug variant with --set-image, or add a busybox container with --target and read the files through /proc/1/root, the pattern covered step by step in debugging distroless containers.

Checking network connectivity from a pod

The ephemeral container shares the pod’s network namespace, so tests run from the pod’s real IP with its real DNS configuration and NetworkPolicies:

kubectl debug -it mypod --image=nicolaka/netshoot --profile=netadmin
~ # dig +search my-service
~ # curl -sv http://my-service.payments.svc:8080/healthz
~ # tcpdump -i any -nn port 5432

If the application sits behind a service mesh sidecar, remember that outbound traffic goes through the proxy; the article on Istio proxy DNS explains why DNS results can differ between the app and a plain debug container.

kubectl debug vs kubectl exec vs Pod Copy

The rule of thumb: use exec if the container is running and has what you need; use an ephemeral container when you need tools the image lacks while observing the live process; use a copy when the container cannot start or you need to change how it starts; use node debugging when the problem is not inside the pod. If you prefer a GUI, desktop clients like Freelens wrap exec and logs, but the three kubectl debug modes remain the most capable option.

Frequently Asked Questions

Conclusion

kubectl debug is three tools behind one command. The ephemeral container mode is what you reach for when a running pod misbehaves and its image has nothing useful in it; the copy mode is for containers that cannot start or need a different command or image; and the node mode replaces SSH for anything below the pod. Set --profile explicitly, grant pods/ephemeralcontainers as carefully as pods/exec, and remember that anything you attach to a live pod stays in its status until the pod is replaced. With those habits in place, a shell-less image stops being an obstacle to debugging.

If your Kubernetes HPA scales down slowly, it is almost certainly doing exactly what it was designed to do. By default the HorizontalPodAutoscaler waits for a 300-second stabilization window before removing pods: it keeps the highest replica recommendation it has computed in the last five minutes, and only drops below that once every recommendation in the window agrees. Add the 15-second sync period, the metrics-server scrape interval and a 10% tolerance band, and “traffic stopped at 10:00, pods started going away at 10:06” is the normal, healthy outcome.

The fix, if you want a different outcome, is the behavior field in autoscaling/v2. This is the short version for “scale down faster”:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60   # default is 300
      policies:
        - type: Percent
          value: 50
          periodSeconds: 30

The rest of this article explains every moving part of HPA scale-down behavior — so you can tell the difference between an HPA that is slow on purpose, one that is stuck, and one that is flapping — and gives tested recipes for each case. If your HPA runs on memory, the memory-specific details (why memory does not fall after load drops, requests vs limits) are covered in Kubernetes HPA on memory; this page is about the scale-down machinery itself, which is the same for every metric.

What Happens Between “Load Dropped” and “Pod Removed”

When people say the HPA is slow to scale down, they usually mean the total elapsed time from the moment load falls to the moment a pod is terminated. That time is the sum of several independent delays, and it helps to see them in order:

1. Metric collection. metrics-server scrapes the kubelets every 15 seconds by default, and the kubelet’s CPU figure is itself a rate over a short window. A drop in load shows up in the resource metrics API somewhere between a few seconds and ~30 seconds later.
2. HPA sync period. The HPA controller in kube-controller-manager evaluates every HPA every 15 seconds (--horizontal-pod-autoscaler-sync-period). Add up to another 15 seconds.
3. Tolerance. If the ratio between the current metric and the target is within 10% of 1.0, the controller does nothing. A gradual decline can spend minutes inside that band.
4. Stabilization window. The controller records a recommendation every sync and, for scale-down, applies the highest recommendation from the last 300 seconds. This is the big one: five minutes minimum after the last high recommendation.
5. Scaling policies. Once the window allows it, the default scale-down policy removes up to 100% of the excess pods every 15 seconds — so this step is fast by default, unless you slowed it down.
6. Pod termination. The Deployment controller deletes pods, which then go through preStop hooks and terminationGracePeriodSeconds. A 60-second graceful shutdown is 60 more seconds before the pod is actually gone.

So with defaults, a clean drop from 10 replicas to 3 typically completes between 5.5 and 6.5 minutes after load falls. If you are seeing 20 minutes, or never, something else is going on — keep reading.

The Default HPA Behavior (and Why Scale-Down Is Deliberately Slow)

When you omit behavior, the HPA behaves as if you had written this (the values are straight from the Kubernetes documentation):

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
      - type: Percent
        value: 100
        periodSeconds: 15
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15
    selectPolicy: Max

The asymmetry is intentional. Scaling up late costs you latency and errors; scaling down late costs you a few minutes of spare capacity. So scale-up reacts immediately (no window, double the fleet or add four pods every 15 seconds, whichever is larger) and scale-down waits five minutes to make sure the drop is real.

The cluster-wide default for the scale-down window comes from the --horizontal-pod-autoscaler-downscale-stabilization flag on kube-controller-manager (5 minutes). On managed platforms — EKS, GKE, AKS — you cannot change controller-manager flags, which is exactly why the per-HPA behavior field exists. Always tune in the HPA object, never rely on the cluster flag.

How stabilizationWindowSeconds Actually Works

The stabilization window is the most misunderstood field in the HPA spec, because the name suggests a delay or a cooldown. It is neither. It is a rolling maximum (for scale-down) or rolling minimum (for scale-up) over the replica recommendations computed during the window.

Concretely: every 15 seconds the controller computes a desired replica count from the metrics and stores it with a timestamp. For scale-down, before acting it looks at all recommendations from the last stabilizationWindowSeconds and uses the highest one. A single high recommendation four minutes ago is enough to hold the fleet at that size for one more minute.

Walk through an example with the default 300-second window:

This is why the scale-down happens in steps that mirror the recommendations from five minutes earlier, and why a brief spike during the quiet period resets the clock. While the window is holding replicas up, kubectl describe hpa shows the condition AbleToScale True ScaleDownStabilized with the message “recent recommendations were higher than current one, applying the highest recent recommendation”. If you see that message, the HPA is not broken — it is waiting.

A few properties of the field worth knowing:

• The value range is 0 to 3600 seconds. Anything above one hour is rejected by API validation.
• stabilizationWindowSeconds: 0 on scale-down means “act on the current recommendation immediately”. Useful for batch workers; dangerous for anything with bursty traffic.
• The same field exists under scaleUp, where it works as a rolling minimum. Setting a scale-up window of 60 seconds means the HPA only scales up if the last minute of recommendations all agree — a good way to ignore one-sample spikes, at the cost of reacting a minute later.
• The window is held in the controller’s memory. When kube-controller-manager restarts or fails over, the recommendation history is lost, and the next scale-down can happen sooner than you expect.

Scaling Policies: Pods, Percent, periodSeconds and selectPolicy

Policies limit how fast the replica count can change once the stabilization window has allowed a change. Each policy says “in any periodSeconds window, change by at most value pods (or value percent of the pods)”.

type: Pods is an absolute number: value: 2 with periodSeconds: 60 means at most two pods removed per minute, regardless of fleet size. This is the predictable option and the one I default to for scale-down.

type: Percent is relative to the replica count at the start of the period: value: 25 with periodSeconds: 60 on a 40-pod fleet allows removing 10 pods in that minute, but on a 4-pod fleet only 1. Percent policies scale with the fleet, which is what you want for large deployments and a nuisance for small ones.

periodSeconds can go up to 1800 (30 minutes). The controller tracks actual scaling events within the period, so a policy of 1 pod per 300 seconds really means “no more than one removal in any rolling five-minute span”, even across multiple HPA syncs.

When you list several policies, selectPolicy decides which one wins:

• Max (the default) picks the policy that allows the biggest change. With Pods: 4 and Percent: 100, a 2-pod fleet can grow by 4 and a 20-pod fleet by 20.
• Min picks the policy that allows the smallest change. This is the conservative choice for scale-down: “at most 10% or 2 pods per minute, whichever is smaller”.
• Disabled turns scaling off in that direction completely. scaleDown.selectPolicy: Disabled means the HPA will only ever add pods.

When a policy is what’s holding the fleet back, kubectl describe hpa shows ScalingLimited True ScaleDownLimit — “the desired replica count is decreasing faster than the maximum scale rate”. That is different from ScaleDownStabilized, and it tells you which knob to turn: the window or the policy.

Tolerance and Rounding: Why the HPA Sometimes Never Scales Down

A frequent complaint is not “the HPA is slow”, but “the HPA never scales down, even though usage is clearly below target”. Two pieces of arithmetic cause most of these cases.

The 10% tolerance band. The controller computes ratio = currentMetricValue / targetValue and skips any action if the ratio is within ±0.1 of 1.0. With a 70% CPU target, an average of 64% gives a ratio of 0.914 — inside the band, so nothing happens. Your fleet can sit at 64% indefinitely, 6 points below target, without a single pod removed.

Ceiling rounding at small replica counts. The desired count is ceil(currentReplicas × ratio). Rounding up is safe for scale-up, but it makes scale-down surprisingly sticky on small fleets. With 3 replicas, a 70% target and an average of 50%:

desired = ceil(3 × 50 / 70) = ceil(2.14) = 3

The ratio (0.71) is well outside the tolerance, yet the result is still 3. To go from 3 to 2 pods, the average must drop to 46.7% or below (2/3 × 70), because those 2 pods would then run at 70%. The HPA is refusing to remove a pod that would push the remaining ones over target — which is correct, but it looks like a bug when you are staring at a dashboard that says 50%. The smaller the fleet, the bigger this effect: going from 2 to 1 requires the average to fall to half the target.

Configurable tolerance per HPA. Until recently the 10% was a cluster-wide constant (--horizontal-pod-autoscaler-tolerance), which on managed clusters meant you could not change it at all. The HPAConfigurableTolerance feature adds a tolerance field per direction:

behavior:
  scaleUp:
    tolerance: 0.05    # react when 5% above target
  scaleDown:
    tolerance: 0.15    # only scale down when 15% below target

The feature went alpha in Kubernetes 1.33, beta in 1.35 (still disabled by default in beta, so managed providers generally did not expose it) and GA in 1.37, where it is always on. On a cluster older than 1.37, check the feature gate before relying on it — an API server that does not know the field silently drops it. A lower scale-up tolerance with a higher scale-down tolerance is a sensible pairing for latency-sensitive services: react early to growth, ignore small declines.

Why an HPA Stops Scaling Down: The Checklist

When the HPA is not scaling down at all — not slowly, not at all — work through these in order. They cover the cases I have actually seen in production.

minReplicas is reached. Obvious, but check it first. ScalingLimited True TooFewReplicas in the conditions confirms it.

Another metric is holding it up. With multiple metrics, the HPA computes a desired count for each and uses the largest. CPU can be at 10% while memory or a queue-length metric keeps the fleet at its current size. kubectl describe hpa lists each metric’s current value; look for the one still near its target. For memory specifically, the reasons it does not fall after load drops are covered in the HPA memory guide.

One metric is failing. This one is subtle and documented: if any metric cannot be fetched and the remaining metrics suggest a scale-down, the controller skips scaling entirely. Scale-up still works in that situation; scale-down does not. A broken Prometheus Adapter query or a missing custom metric can therefore freeze your fleet at its peak size for days. Look for FailedGetPodsMetric, FailedGetExternalMetric or FailedGetResourceMetric events.

Pods with missing metrics. When some pods have no metrics yet (new pods, or pods on a node where metrics-server is failing), the controller recomputes conservatively and assumes those pods are at 100% of target when evaluating a scale-down. A handful of pods with no metrics can keep the average above the scale-down threshold.

Rounding on a small fleet. See the previous section. On 2 or 3 replicas, the average may need to fall far below the target before a pod is removed.

Scale-down is disabled. Someone may have set scaleDown.selectPolicy: Disabled, or a very long window. kubectl get hpa <name> -o yaml and read the behavior block — including defaults, which the API server fills in.

Something else owns the replica count. A GitOps tool that sets spec.replicas on the Deployment, a KEDA ScaledObject targeting the same Deployment, or a second HPA will fight the HPA. ArgoCD will even show the Deployment as out of sync every time the HPA changes it. Remove replicas from the manifest you apply, and make sure exactly one autoscaler targets each workload.

Requests are wrong. Utilization is usage divided by request. If requests are set far below real idle usage, “idle” is already above target and the HPA will never scale down. The fix is to right-size requests, not to tune the HPA — see Kubernetes resource requests and limits.

Why an HPA Scales Down Too Fast (or Flaps)

The opposite problem shows up on bursty workloads: the HPA removes pods, traffic comes back two minutes later, the HPA adds them again, and new pods take a while to be useful. Every cycle costs cold starts, connection re-balancing and, often, a latency spike.

Common causes:

• A short or zero scale-down window. Teams set stabilizationWindowSeconds: 0 to “fix” slowness and create flapping. If traffic has a periodicity shorter than your window — a cron that fires every 10 minutes, a batch that arrives every quarter hour — the window must be longer than that period.
• Aggressive Percent policies. Percent: 100 allows removing all excess pods at once. A Pods policy that removes one or two per period turns a cliff into a staircase, which gives you time to notice a returning load.
• Slow-starting pods. If a new pod takes three minutes to warm up — JVM JIT compilation, cache loading, connection pools — every scale-down you later have to reverse costs three minutes of under-capacity. The HPA has two cluster-level safeguards for CPU (--horizontal-pod-autoscaler-cpu-initialization-period, 5 minutes, and --horizontal-pod-autoscaler-initial-readiness-delay, 30 seconds) that ignore CPU samples from pods that are still starting. They only help if your readiness probe does not report Ready before the warm-up is done. Use a startupProbe that holds the pod back until it can actually serve.

HPA Behavior Recipes (autoscaling/v2)

All of these are complete manifests you can apply as-is after changing the target and metrics. They use CPU for brevity; the behavior block is the part that matters, and it works identically with memory, custom or external metrics.

Scale down faster

For stateless services with fast startup and load that drops cleanly — internal APIs, dev and staging environments, anything where idle capacity costs more than an occasional cold start:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-fast-down
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 50
          periodSeconds: 30
      selectPolicy: Max

The fleet can shrink one minute after load drops, halving at most every 30 seconds. Going below 60 seconds rarely buys anything real: metric lag already eats 15-30 seconds of it.

Scale down gradually (fast up, slow down)

The safe default for customer-facing services, bursty traffic and slow-starting runtimes:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-gradual
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
        - type: Pods
          value: 4
          periodSeconds: 30
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Pods
          value: 2
          periodSeconds: 120
        - type: Percent
          value: 10
          periodSeconds: 120
      selectPolicy: Min

Scale-up doubles the fleet (or adds 4, whichever is more) every 30 seconds. Scale-down waits until ten minutes of recommendations agree, then removes at most 2 pods or 10% every two minutes, whichever is smaller. A drop from 50 to 10 pods takes about 40 minutes, which is exactly the point: if traffic comes back, you are still mostly provisioned.

Never scale down automatically

For workloads where the HPA is there to absorb peaks, and scale-down should happen only through a deploy or a human decision — stateful consumers that rebalance partitions on every change, services with very expensive warm-up, or the first weeks of a new HPA you do not yet trust:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: consumer-no-down
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: consumer
  minReplicas: 4
  maxReplicas: 24
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      selectPolicy: Disabled

The cost: the fleet sits at its high-water mark until someone lowers maxReplicas or redeploys.

KEDA: cooldownPeriod Is Not What Controls Scale-Down

If you use KEDA, you have probably tried cooldownPeriod to slow down or speed up scale-down and seen no effect. That is because cooldownPeriod (default 300 seconds) only applies to the transition from 1 replica to 0: how long KEDA waits after the last active trigger before scaling the workload to zero. Every change between N and 1 replicas is made by the HPA that KEDA creates, and is controlled by the same behavior field described above — passed through the ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker
spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 0
  maxReplicaCount: 40
  cooldownPeriod: 300          # only 1 -> 0
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:                # everything between maxReplicaCount and 1
        scaleDown:
          stabilizationWindowSeconds: 120
          policies:
            - type: Percent
              value: 25
              periodSeconds: 60
  triggers:
    - type: rabbitmq
      metadata:
        queueName: jobs
        mode: QueueLength
        value: "20"
      authenticationRef:
        name: rabbitmq-auth

For queue-driven workers, a short window is usually fine: queue length is a leading signal, not a lagging one like CPU. More on KEDA triggers in event-driven autoscaling with KEDA.

How to Diagnose HPA Scale-Down with kubectl

Three commands tell you almost everything.

Watch the HPA live to see the relationship between the metric and the replica count over time:

kubectl get hpa api -w

Describe it to see the conditions and the recent events — this is where the “why” lives:

kubectl describe hpa api

A healthy HPA waiting on its stabilization window looks like this:

Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  18% (90m) / 70%
Min replicas:                                          2
Max replicas:                                          30
Behavior:
  Scale Up:
    Stabilization Window: 0 seconds
    Select Policy: Max
    Policies:
      - Type: Pods     Value: 4    Period: 15 seconds
      - Type: Percent  Value: 100  Period: 15 seconds
  Scale Down:
    Stabilization Window: 300 seconds
    Select Policy: Max
    Policies:
      - Type: Percent  Value: 100  Period: 15 seconds
Deployment pods:       10 current / 10 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range

How to read the conditions:

If ScalingActive is False, stop tuning behavior: the HPA is not receiving metrics, and the events at the bottom of the describe output (FailedGetResourceMetric, FailedComputeMetricsReplicas) say why.

Check the raw metrics the HPA sees, to rule out metrics-server lag or missing pods:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods" | jq '.items[] | {name: .metadata.name, cpu: .containers[].usage.cpu}'
kubectl top pods -l app=api

If some pods are missing from that output, you have found your “missing metrics treated as 100%” problem.

Pod Scale-Down Is Not Node Scale-Down

One last source of confusion: even when the HPA removes pods promptly, the nodes do not disappear at the same time, so the cloud bill does not drop when you expect. Node removal is a separate decision made by Cluster Autoscaler or Karpenter, with its own delays (Cluster Autoscaler waits 10 minutes by default before removing an underutilized node) and its own blockers, such as PodDisruptionBudgets and pods that cannot be moved. If cost is the reason you want faster scale-down, tune both layers — the comparison of how each one consolidates nodes is in Cluster Autoscaler vs Karpenter.

And before tuning behavior at all, make sure the metric is the right one: a CPU target on a service that is really bound by I/O or a downstream dependency will never scale down cleanly, no matter what the window is. The broader design questions — which metric, which target, when HPA is the wrong tool — are in Kubernetes HPA best practices.

Frequently Asked Questions

Conclusion

A Kubernetes HPA that scales down slowly is, nine times out of ten, the default 300-second stabilization window doing its job, plus metric and sync lag on top. Treat that as a starting point, not a bug: shorten the window for services that start fast and cost money when idle, lengthen it and add a Pods policy for services where every cold start hurts, and disable scale-down entirely where only a human should make that call.

When the HPA does not scale down at all, stop tuning behavior and read kubectl describe hpa. The conditions tell you whether you are looking at stabilization (ScaleDownStabilized), a rate limit (ScaleDownLimit), a floor (TooFewReplicas) or broken metrics (ScalingActive False) — and the tolerance and rounding arithmetic explains the rest. Get that right, and the HPA becomes predictable enough that you can reason about its next move instead of waiting for it.

Every week brings another “we built a Kubernetes-native X” announcement. Some of those projects will be running your production clusters in three years; most will be archived repositories with a “no longer maintained” banner. The CNCF maturity level — Sandbox, Incubating, Graduated — is the most widely used shortcut for telling the two apart, and it is also one of the most misread.

The labels are not marketing tiers. Each one corresponds to a documented set of criteria that the Technical Oversight Committee (TOC) checks in a due-diligence process, and each one is a floor, not a guarantee. This article goes through what the TOC actually requires at each level as of 2026, what the levels do not tell you, the 2025–2026 events that show the ladder moving in both directions, and a practical framework for turning a badge into an adoption decision.

The Short Answer

Sandbox means the TOC judged the idea worth hosting; it says almost nothing about stability, security posture or whether anyone will maintain it next year. Incubating means the project has documented governance, at least three independent production adopters interviewed by the TOC, a security self-assessment and a passing OpenSSF badge — it is the level at which the CNCF starts actively evaluating adoption. Graduated adds maintainers from at least two organisations, an org-balance mechanism in governance and a completed third-party security audit — it is the level at which the foundation is willing to say the project has demonstrated production readiness at scale.

If you need one rule: Sandbox is where you experiment, Incubating is where you can build with a contingency plan, Graduated is where you standardise.

What the TOC Actually Requires

The criteria live in the TOC repository, and since 2024 the detailed lists sit inside the incubation and graduation application templates rather than a single criteria document. These are the items that matter for an adopter, quoted from the current templates.

Three things in that table are worth reading twice.

First, Sandbox has almost no technical bar. The published reasons applications get closed without review are administrative: not Apache 2.0, no MAINTAINERS file with employer affiliations, being a reference architecture rather than a reusable project, or being a subproject without the parent’s consent to split. Organisation diversity is explicitly “not a requirement, but considered during review”. The TOC reviews applications in a non-public session roughly every two months, working through seven to ten at a time in first-in-first-out order. There is no security review, no adopter interview and no architecture review at this stage — TAG-led General Technical Reviews are optional and “do not result in a label or a required action for applicants”.

Second, the OpenSSF badge requirement is “passing” at both Incubating and Graduated, not silver or gold. If a project advertises a gold badge, that is the project going beyond the bar, not the CNCF demanding it.

Third, the third-party security audit only arrives at Graduation. An Incubating project has assessed itself. That is not nothing — the self-assessment template is thorough and TAG Security reviews it — but it is not an external audit of the code you are about to run with cluster-admin privileges.

What the Levels Do Not Mean

Sandbox is not an endorsement

The lifecycle document describes Sandbox projects as ones where “failure is a possibility” and that “are expected to undergo significant changes, including breaking changes to their functionality”. It adds that they “might still be used in production by a few organizations on a case to case basis” — which is the TOC’s careful way of saying that it is possible, and that the risk is entirely yours.

The practical consequence: a Sandbox badge on a README tells you the licence is clean and the maintainers are identifiable. It does not tell you the API will exist in the same shape next quarter.

Incubating is not “almost graduated”

Projects can sit at Incubating for a very long time. Thanos entered the CNCF in July 2019 and moved to Incubating on 19 August 2020; six years later it is still Incubating and still perfectly healthy. Contrast KEDA — Sandbox March 2020, Incubating August 2021, Graduated August 2023 — or Kyverno, which took from November 2020 to March 2026 to make the full journey. Time at a level is a function of governance work and audit scheduling as much as technical maturity, so “still Incubating” is not a warning sign on its own.

Graduated is not “right for you”

Graduation says the project has cleared the foundation’s bar for governance, security and adoption breadth. It does not say the project fits your scale, your team or your problem. Cilium, Argo, Kyverno and KEDA are all Graduated; you should still not run all four if you have one cluster and three engineers. The badge removes the “will this be abandoned” question from your evaluation; it does not answer any of the others.

And some things you assume are CNCF projects are not

Karpenter is a good example of the trap. It is hosted under kubernetes-sigs as a subproject of SIG Autoscaling, which means it is governed by the Kubernetes project, not by the CNCF TOC as a standalone project with its own maturity level. It has no Sandbox, Incubating or Graduated badge to check, and any blog post that assigns it one is guessing. When you evaluate it, you are evaluating a Kubernetes SIG subproject, which is a different (and generally stricter) governance regime. The Cluster Autoscaler vs Karpenter comparison covers the technical side.

The Ladder Moves in Both Directions

2026 has been an unusually busy year for graduations: Dragonfly (January), Kyverno (16 March, announced 24 March), OpenTelemetry (May), Cloud Native Buildpacks (August), Kubeflow (August) and Karmada (8 September). Karmada is the clean illustration of the full path — Sandbox September 2021, Incubating December 2023, Graduated September 2026, with the announcement explicitly noting that the project “completed a third-party security audit, established a formal steering committee to ensure transparent governance” to get there.

The archive side is less publicised and more instructive. In 2025 the TOC archived two batches of Sandbox projects: Nocalhost, SuperEdge and KubeDL in March, then Merbridge, Sealer, Teller, DevStream and OpenELB in May. All eight were Sandbox projects, several of them with a single corporate sponsor, and each went through the documented process — a project-health issue, a two-week final comment period, then a two-week TOC vote requiring a 2/3 majority.

Archiving is also not necessarily the end. OpenEBS was archived after a TOC vote in early 2024 and re-admitted to Sandbox in October 2024 with a rebuilt maintainer team, because “any project can be reactivated into CNCF by following the normal Project Lifecycle & Process”. What an archived project loses is real, though: it “may not be used any more” to advertise its former status, and it disappears from the landscape, DevStats and CLOMonitor.

For an adopter the lesson is simple. Sandbox is the level where the foundation has the least information about the project and the least reason to fight for it. If the project has one employer behind its maintainers, the foundation’s exit process is the one you should be planning for.

Signals You Can Read Yourself

The TOC’s own inputs for health and archiving decisions are public, so you can run the same checks before the TOC does. The archiving policy lists the signals it weighs: the project health dashboard, whether the project still meets current acceptance criteria, community requests, “security responsiveness”, meeting activity, mailing-list engagement and adoption statistics.

Project Health issues. Concerns are filed in cncf/toc with the [HEALTH] prefix. The template describes its purpose as ascertaining “the current activity and health of the project so the TOC may identify the appropriate support and guidance for the project to return to an optimal state of health or determination of archival”. Search the TOC issues for the project name before you adopt; a health issue is a leading indicator by months.

Level-change applications. Incubation and graduation applications are also public issues in cncf/toc. Reading one tells you exactly what the TOC asked, which adopters were interviewed and what was flagged in the security review. This is better due diligence than most vendor RFP responses.

DevStats and LFX Insights. DevStats (devstats.cncf.io) gives contributor and company counts over time; LFX Insights adds organisational affiliation. The metric to watch is not stars but the number of companies contributing in the last year and whether that number is rising.

CLOMonitor. clomonitor.io scores every CNCF project against the same checklist the TOC uses — licence, governance files, security policy, OpenSSF badge, DCO, release signing. A low score at Incubating is worth a question; a low score at Sandbox is normal.

TAG structure. In 2025 the TOC rebooted its Technical Advisory Groups into five: Developer Experience, Infrastructure, Operational Resilience, Security and Compliance, and Workloads Foundation. TAGs no longer perform formal reviews as part of the Sandbox process; that work moved to the TOC’s Project Reviews subproject. If you are reading an old TAG review of a project, check the date — the group that wrote it may no longer exist.

A Practical Evaluation Framework

The maturity level is one input. Here is the checklist that turns it into a decision, with the commands and URLs to answer each question in under an hour.

1. Where does it sit in your stack?

A CNI plugin, a policy engine or a service mesh is foundational: replacing it is a migration project. A dashboard, a CLI plugin or a diagram generator is peripheral: replacing it is an afternoon. The same badge means different things at different layers. Headlamp is a Sandbox project (accepted 17 May 2023) and a perfectly reasonable choice as a cluster UI, because a UI is peripheral — see Headlamp vs FreeLens vs Lens. A Sandbox CNI is a different conversation.

2. Who actually maintains it?

# Committers in the last year, with email domains
git shortlog -sne --since="1 year ago" | head -20

# How concentrated is it? Count distinct email domains among the top 10
git shortlog -sne --since="1 year ago" | head -10 \
  | grep -oE '@[a-z0-9.-]+' | sort | uniq -c | sort -rn

If one domain accounts for more than about 80% of commits, the project’s roadmap is that company’s roadmap, regardless of what the governance document says. This is exactly the “org-balance” question the TOC asks at Graduation; ask it earlier.

3. What does the security posture look like?

# OpenSSF Scorecard: branch protection, signed releases, dependency update tooling, fuzzing
scorecard --repo=github.com/<org>/<project>

Then check for SECURITY.md, whether reported CVEs have advisories with fix versions, and whether release artefacts are signed. For an Incubating project, read the Security Self-Assessment linked from its application issue; for a Graduated one, find the third-party audit report — it is usually published in the project repository or by the auditor.

4. What is the release and compatibility story?

Look at the last twelve months of releases: cadence, whether minor releases carry breaking changes, and whether there is a written deprecation policy. “Fewer breaking changes and more stable, versioned APIs” is literally the TOC’s description of the Incubating transition, so a Sandbox project with a stable API is ahead of its level and an Incubating project without one is behind it.

5. Who else depends on it?

A project that is a dependency of other CNCF projects — or that a managed Kubernetes service ships by default — has a survival guarantee no badge can provide. Cilium is Graduated, but the more important fact for its longevity is that several cloud providers build their networking on it.

6. What is the exit?

Write down, before adopting, what it would take to replace the project. For a policy engine: how many policies, how portable are they (Kyverno’s YAML versus Rego, for example — see enforcing standard and custom policies with Kyverno). For a UI: nothing. For a storage layer: everything. The lower the level, the more this answer matters.

Mapping Level to Adoption Strategy

Start here:
│
├── Is the component foundational (network, storage, policy, identity, mesh)?
│   ├── YES → Graduated by default.
│   │         Incubating only with a documented exit plan and ≥2 maintainer orgs.
│   │         Sandbox: no.
│   └── NO ↓
│
├── Is it core to delivery (CI/CD, GitOps, autoscaling, observability pipeline)?
│   ├── YES → Incubating or Graduated.
│   │         Sandbox only if you are willing to fork or contribute fixes.
│   └── NO ↓
│
├── Is it peripheral (UI, CLI plugin, docs/diagram tooling, developer convenience)?
│   ├── YES → Any level; evaluate on fit and maintainer count, not badge.
│   └── NO ↓
│
└── Not a CNCF project at all (Kubernetes SIG subproject, vendor OSS, no foundation)?
    → Run the same six checks; the absence of a badge is information, not a verdict.

A few corollaries that follow from the criteria rather than from opinion:

• Sandbox projects deserve a re-evaluation date. Put it in the calendar at twelve months. If the project has not applied for Incubation, or has a [HEALTH] issue open, that is when you decide whether to contribute or leave.
• Incubating is the sweet spot for most platform teams. The governance and security documentation exists, adopters have been interviewed, and the project is still moving fast enough to fix the thing you need. Most of what you will evaluate this year sits here.
• Graduated projects earn platform-wide mandates. They are the ones where investment in training, internal documentation and deep expertise has a long-term return, because the foundation has structurally reduced the abandonment risk.
• When a project moves level, re-run the checklist, not the decision. Graduation does not change your architecture. Archiving does — it starts the clock on your exit plan, and the two-week comment period on the archive issue is where you find out whether anyone intends to fork.

Frequently Asked Questions

Conclusion

The CNCF maturity ladder is the best free due diligence in the industry, but only if you read it as what it is: a record of which documented criteria a project has satisfied, checked by a committee that publishes its reasoning. Sandbox tells you the licence is clean. Incubating tells you adopters have been interviewed and governance exists. Graduated tells you an outside party has audited the code and no single company can walk away with it.

None of the three tells you whether the project fits your stack, and none of them replaces the six checks above. Run them, write down the exit, and re-run them when the level changes — in either direction.

FreeLens Extensions: The Short Answer

FreeLens installs extensions from npm. Open the Extensions page (⌘⇧E on macOS, Ctrl+Shift+E on Linux and Windows), paste the package name — for example @freelensapp/resource-map-extension — and press Install. There is no curated in-app marketplace yet, so the package name is the address: you get it from the extension’s repository or its npm page.

Related reading: FreeLens overview and installation.

Two things trip up almost everyone arriving from OpenLens. First, @freelensapp/freelens-node-pod-menu is deprecated and now ships as an empty package — installing it does nothing, because that functionality moved into FreeLens itself. Second, OpenLens and Lens extensions are not drop-in compatible; several have been ported, but you install the FreeLens port, not the original.

This catalogue is current as of FreeLens v1.10.3 (July 2026). If you are still deciding which client to standardise on, see FreeLens vs OpenLens vs Lens.

How to Install a FreeLens Extension

1. Open FreeLens and go to File → Extensions, or press ⌘⇧E / Ctrl+Shift+E.
2. Paste the npm package name into the input field — the scoped name in full, including the @freelensapp/ prefix where it applies.
3. Press Install. FreeLens downloads the package from the npm registry and unpacks it.
4. Wait for the extension to appear in the installed list with its Enabled toggle on. Some extensions add a new entry to the left sidebar; others add menu items to existing resource views, so a missing sidebar icon does not mean the install failed.

Extensions are per-user, not per-cluster: once installed, an extension is available for every cluster in your catalogue. If an extension needs CRDs that a given cluster does not have — the Karpenter or FluxCD extensions, for instance — its views simply stay empty on that cluster rather than erroring.

Official FreeLens Extensions

These live under the freelensapp GitHub organisation and are maintained alongside the app itself.

If you run Karpenter, the extension pairs naturally with the trade-offs covered in Karpenter vs Cluster Autoscaler — seeing NodeClaims appear and consolidate in real time makes the disruption behaviour much easier to reason about than reading events.

Community Extensions

Maintained by individuals outside the core organisation. They are genuinely useful, but check the version number before you rely on one in a work setting — several are still 0.x.

Spotting Expiring Tokens, Certificates and Stale Secrets

This is the single most common thing people go looking for an extension to solve, and stock FreeLens does not do it: the Secrets view shows you that a Secret exists, not whether the thing inside it has expired. A TLS certificate three days from expiry looks exactly like one with a year left.

Credential Explorer (freelens-credential-explorer) is the extension aimed squarely at that gap. It builds a dashboard around the questions you actually ask during an incident: how old is this Secret, when does it expire, which namespace is it in, and what is consuming it — following the chain from Secret through Deployment and Pods to the ServiceAccount. It recognises credentials from Vault, External Secrets, Argo CD, cert-manager and TLS certificates, Helm releases, and plain Opaque secrets.

Two caveats worth knowing before you install it: it declares itself a proof of concept at 0.1.x, and it requires FreeLens v1.9.0 or later. For a cluster where certificate expiry is a genuine production risk, treat it as a fast triage view rather than as your alerting — the durable answer is a cert-manager alert or a Prometheus rule on certmanager_certificate_expiration_timestamp_seconds, which pages you at three in the morning when a desktop IDE cannot.

Why freelens-node-pod-menu Does Nothing Any More

Early FreeLens shipped without the node and pod context menus that OpenLens users expected — shell into a pod, cordon a node, view logs from the right-click menu. The gap was filled by a separate extension, @freelensapp/freelens-node-pod-menu, and just about every migration guide written in 2025 tells you to install it.

That functionality has since been absorbed into FreeLens itself. The package still exists on npm and still installs cleanly, but it is now an empty package that provides nothing. If you followed an older guide and are wondering why the extension appears installed yet contributes no visible feature, this is why — and the menus you were after are already there in the core app. Uninstall it and move on.

Can You Install OpenLens or Lens Extensions?

Not directly. FreeLens forked from OpenLens, so the extension API is closely related and porting is usually mechanical, but the packages are published separately and an original Lens extension is not guaranteed to load. In practice a good number of the popular OpenLens extensions have already been ported — the resource map is the obvious example — and the pattern is to look for a freelens- flavoured package rather than to force the original.

If the extension you depend on has no port, forking and adapting it is a realistic afternoon of work where the licence permits, and the project publishes an example extension as a starting template. That is a meaningful difference from Lens Desktop, where the licensing of the binaries is the constraint rather than the code.

When an Extension Will Not Install

• Nothing happens after pressing Install. Check the package name character for character — the scope matters, and @freelensapp/resource-map-extension and freelens-resource-map-extension are not the same string. The npm name is authoritative, not the GitHub repository name.
• The extension installs but its views are empty. Almost always missing CRDs on that cluster. The FluxCD, Karpenter, Gateway API, Sveltos and Kamaji extensions all render nothing useful against a cluster that does not run the corresponding controller.
• It installed but added no sidebar entry. Not every extension adds one; some only extend existing detail panes and context menus.
• It worked before an upgrade and now does not. Community extensions track FreeLens releases loosely. Pin your FreeLens version if an extension is load-bearing for your workflow, and check the extension’s declared minimum version before upgrading the app.

Frequently Asked Questions

Further Reading

• FreeLens vs OpenLens vs Lens — which of the three to standardise on, and what changed after Mirantis restricted the Lens binaries.
• Headlamp vs FreeLens vs Lens — the CNCF option, and when a web-based UI beats a desktop app.
• Kubernetes Dashboard alternatives in 2026 — the wider field, including k9s and Headlamp.
• Radar — a newer entrant built around topology and a persistent event timeline.

If you manage Kubernetes in production, you’ve likely felt the sting of a feature that promised stability but delivered chaos. The community is rich with stories of PodDisruptionBudgets (PDBs) that blocked critical updates, misconfigured liveness probes that created restart loops, and resource limits that turned a simple deployment into a cascading failure. These aren’t inherently “bad” features; they are powerful tools that, like a surgeon’s scalpel, require precise understanding and context to use effectively.

Related reading: liveness probe anti-patterns.

Drawing from a wealth of shared experience—including community discussions and documented failure stories—a clear pattern emerges. The gap between a feature’s theoretical promise and its production reality is often bridged not by more documentation, but by operational rigor. This post analyzes common pitfalls, not to discourage the use of these features, but to provide a decision framework for platform teams to evaluate adoption, focusing on observability, gradual rollout, and clear rollback plans.

The Gap Between Theory and Practice: Features That Bite Back

Kubernetes is designed to automate complex distributed systems patterns. However, this automation can amplify misconfigurations at scale. The following features are frequently cited as sources of production pain, precisely because their power is double-edged.

1. PodDisruptionBudgets (PDBs): The Update Blocker

On paper, a PDB is a safeguard. It ensures a minimum number of pods for a critical application remain available during voluntary disruptions like node drains or cluster upgrades. The theory is flawless.

The practice, as shared by many engineers, reveals the trap: a PDB with overly restrictive minAvailable or maxUnavailable settings can completely halt cluster maintenance. Imagine a deployment with 3 pods and a PDB set to minAvailable: 3. Any drain operation is now impossible, stalling node security patches or Kubernetes version upgrades. The cluster’s ability to heal and evolve is held hostage by a configuration intended to protect it.

The deeper lesson isn’t to avoid PDBs, but to configure them with the system’s evolution in mind. They must allow for the cluster’s own lifecycle operations.

2. Liveness and Readiness Probes: The Self-Inflicted Outage

Probes are the cornerstone of Kubernetes’ self-healing and traffic management. A liveness probe failure restarts the pod; a readiness probe failure removes it from service endpoints. This is essential for resilience.

In production, misconfigured probes are a classic source of instability. Common pitfalls include:

• Overly sensitive liveness checks: A probe checking an endpoint that briefly spikes in latency due to a downstream cache miss can cause a restart loop, exacerbating the problem and taking the service fully down.
• Resource-intensive probes: A probe that executes a heavy database query every few seconds can itself become a source of resource exhaustion and latency, creating a feedback loop of failure.
• Incorrect readiness signals: An application marked “not ready” during its entire startup or lengthy initialization will never receive traffic, appearing as a deployment failure.

As noted in the Kubernetes configuration overview, probes must be designed to reflect the actual health of the application, not an idealized state. They should be cheap, stable, and representative.

3. Resource Requests and Limits: The Silent Strangulation

Setting CPU and memory requests/limits is Kubernetes 101. They ensure fair scheduling and prevent a single pod from consuming all node resources. The theory is fundamental to multi-tenancy.

The production reality is subtler. Setting limits too low (“limit starvation”) is a frequent cause of mysterious, intermittent failures. A pod hitting its CPU limit is throttled, causing increased latency and timeouts. A pod hitting its memory limit is OOMKilled instantly. The symptoms—slow responses or disappearing pods—often point to application bugs, masking the true infrastructure cause.

Conversely, setting requests too high leads to poor cluster utilization and scheduling headaches. The key is continuous observation: limits should be informed by actual usage under load, not initial guesses.

4. Helm Hooks and Complex Operators: The Unpredictable Orchestrator

Helm hooks and custom operators automate complex lifecycle tasks: database migrations, secret injection, or pre-upgrade validation. They abstract away imperative steps.

In production, this abstraction can become a black box. A post-install hook that fails can leave a release in a stuck state. An operator with a bug in its reconciliation logic can enter a loop, endlessly creating and deleting resources. The complexity of debugging an automated system that has gone awry often far exceeds the complexity of the manual process it replaced. The failure stories aggregated in resources like kubernetes-failure-stories are replete with examples of automation gone wrong.

A Framework for Safer Feature Adoption

Banning powerful features is not the answer. The goal is to adopt them with eyes wide open. Platform teams should implement a framework that evaluates risk and mandates safeguards. Here is a practical, four-phase approach.

Phase 1: Evaluation & Contextual Understanding

Before enabling a feature cluster-wide or recommending it to application teams, ask:

• What problem does this solve for us? Is it a real pain point, or just a “nice-to-have”?
• What is the failure mode? How can this feature break? (e.g., PDBs block drains, probes cause restarts).
• What are the observability requirements? What metrics, logs, and alerts do we need to see if it’s misbehaving?
• What is the rollback procedure? How do we quickly disable or revert this feature if it causes an incident?

Phase 2: Implementation with Guardrails

Deploy the feature with constraints that limit its blast radius.

• Start with non-critical workloads: Apply PDBs first to staging or low-priority services.
• Use sane defaults via Policy-as-Code: Use tools like OPA/Gatekeeper or Kyverno to enforce safe defaults. For example, a policy could forbid PDBs with minAvailable: 100% or enforce a maximum probe timeout.
• Document the “why” and the “how to escape”: Annotate resources or maintain runbooks that explain the configuration and the steps to neutralize it in an emergency.

Phase 3: Gradual Rollout & Observability

Treat feature adoption like a software deployment.

• Canary the configuration: Apply a new PDB or aggressive probe to one pod or one namespace first. Monitor its effect closely.
• Implement specific monitoring: Beyond general cluster health, create alerts for:
  – PDBs blocking evictions for > X minutes.
  – Pod restart counts spiking (potential probe issue).
  – Containers hitting CPU throttling or being OOMKilled.
  – Helm releases stuck in a pending hook state.

A simple Prometheus alert for PDB blockage might look like this:

# Alert if a PDB is blocking voluntary pod disruptions for too long
- alert: PDBBlockingDisruption
  expr: kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy
    and (kube_poddisruptionbudget_status_desired_healthy - kube_poddisruptionbudget_status_expected_pods) == 0
    and kube_poddisruptionbudget_status_disruptions_allowed == 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "PDB {{ $labels.namespace }}/{{ $labels.poddisruptionbudget }} is blocking all pod disruptions"
    description: "The PDB requires all pods to be available, preventing node drains or updates for 10 minutes."

Phase 4: Review and Iteration

Adoption isn’t a one-time event. Regularly review:

• Are the features providing the intended value? Are PDBs actually increasing availability during updates?
• What incidents or near-misses have they been involved in? Use post-incident reviews to refine configurations and policies.
• Can we improve defaults or abstractions? Can the platform team provide a simplified, safe Custom Resource or Helm chart that encapsulates best practices?

Frequently Asked Questions

Conclusion: Embrace Power, Respect Complexity

The history of engineering is the history of building more powerful tools and learning to wield them safely. Kubernetes features like PDBs, probes, and resource management are no different. Their potential for causing production pain is a direct reflection of their power to automate complex, critical behaviors.

The path forward is not avoidance, but disciplined adoption. By shifting from a mindset of “enable and hope” to a framework of “evaluate, guard, observe, and iterate,” platform teams can harness these powerful features to build more resilient, self-healing systems without becoming victims of their own automation. The lessons are already written in the community’s failure stories; the task is to learn from them and build a safer, more informed practice.

If you manage multi-tenant Kubernetes clusters or enforce strict separation between development, staging, and production environments, you likely rely on namespaces as a primary security boundary. It’s a logical, clean model: workloads and users are isolated within their designated namespaces, creating a clear perimeter for access control and network traffic.

Related reading: the 2026 Kubernetes hardening guide.

But what if this boundary is more of a suggestion than a hard wall? The reality is that namespace isolation, as defined by the Kubernetes security model, is a construct built on the correct configuration of several underlying controls. When one of those controls fails—often through misconfigured platform-level tools—the entire boundary can collapse, enabling lateral movement that compromises the security of your entire cluster.

This isn’t a theoretical concern. Recent vulnerabilities, like the one detailed in a discussion on CVE-2026-22039, demonstrate how admission controllers, which are meant to enforce security, can be exploited to bypass namespace isolation entirely. This event isn’t an anomaly; it’s a symptom of a broader pattern where the complexity of the platform layer introduces critical gaps in a core security tenet.

The Illusion of the Hard Boundary

Kubernetes documentation is clear: “Namespaces are a way to divide cluster resources between multiple users.” They are a mechanism for scoping names and organizing objects. However, they are not a security feature by themselves. Isolation is achieved through the combination of:

• RBAC (Role-Based Access Control): Governing who can do what, and where.
• Network Policies: Controlling pod-to-pod communication.
• Admission Controllers: Validating and mutating requests before persistence.
• Resource Quotas & Limit Ranges: Managing resource consumption.

The security boundary exists only when all these layers are correctly configured and aligned. A flaw in any one layer—especially in a cluster-scoped component like an admission controller, a monitoring agent, or a service mesh sidecar injector—can create a bridge between namespaces.

How the Boundary Fails: Real-World Exploit Paths

Let’s examine common failure modes that break namespace isolation, moving from conceptual to concrete.

1. Privileged Admission Controller Exploits

The CVE-2026-22039 discussion highlights a classic case. An admission controller with broad permissions (e.g., cluster-admin or powerful ClusterRole bindings) is deployed to validate resources. If this controller has a vulnerability—such as improperly validating the requesting user or namespace of the mutated object—an attacker could craft a request that tricks the controller into creating or modifying resources in a namespace they should not have access to.

Attack Flow:

1. Attacker has compromised a pod in the tenant-a namespace.
2. They discover a cluster-scoped admission controller (e.g., a policy engine) is vulnerable to a confused deputy attack.
3. They send a malicious payload to the API server that triggers the admission controller.
4. The admission controller, operating with high privileges, is tricked into creating a privileged ServiceAccount or a pod with host network access in the kube-system namespace.
5. Isolation is broken; lateral movement to a critical namespace is achieved.

2. Misconfigured Network Policies (or Their Absence)

By default, Kubernetes networking allows all pods to communicate with each other, regardless of namespace. Without Network Policies, a compromised pod in dev can directly probe and attack pods in production. Even with policies, a single overly permissive rule (e.g., allowing ingress from all namespaces for a debugging port) can create a breach.

3. Over-Permissioned Service Accounts & Pods

ServiceAccounts are namespaced, but the RBAC Roles or ClusterRoles bound to them are not. A common misconfiguration is binding a namespaced ServiceAccount to a powerful ClusterRole. A pod using that ServiceAccount effectively has those cluster-wide privileges, allowing it to read secrets, delete pods, or create bindings in any namespace.

A Framework for Testing Namespace Boundaries

Assuming isolation is dangerous. You must actively test it. Here is a practical framework for platform and security teams.

Phase 1: Discovery & Mapping

• Inventory Cluster-Scoped Components: List all deployments, daemonsets, and pods in kube-system, gatekeeper-system, istio-system, etc. Document their assigned ServiceAccounts and associated RBAC.
• Audit RBAC Bindings: Use kubectl get clusterrolebindings -o wide and kubectl get rolebindings --all-namespaces to find any bindings of powerful roles to ServiceAccounts or users in non-critical namespaces.
• Map Network Policies: Generate a visual or logical map of allowed ingress/egress flows between namespaces. Identify namespaces with no policies applied.

Phase 2: Active Penetration Testing

From the perspective of a compromised pod in a non-privileged namespace (simulate with a benign test pod), run controlled tests.

Test for Privilege Escalation:

# Inside the test pod
# 1. Check the pod's own permissions
kubectl auth can-i --list

# 2. Attempt to list resources in other namespaces
kubectl get pods -n kube-system
kubectl get secrets -n production

# 3. Attempt to create a pod in another namespace
cat <<EOF | kubectl apply -f - --namespace=kube-system
apiVersion: v1
kind: Pod
metadata:
  name: test-breakout
spec:
  containers:
  - name: busybox
    image: busybox
    command: ['sh', '-c', 'sleep 3600']
EOF

Test for Network Access:

# Use netcat or curl to probe internal services in other namespaces
# Find the ClusterIP of a service in the target namespace
kubectl get svc -n production --output=wide

# From test pod, attempt to connect
curl -v http://<production-service-cluster-ip>:<port>
nc -zv <production-service-cluster-ip> <port>

Phase 3: Admission Controller Stress Testing

This is more advanced but critical. Craft resource manifests designed to probe the logic of your policy engines (e.g., OPA Gatekeeper, Kyverno, custom webhooks).

• Submit requests with mismatched or spoofed namespace fields in the object metadata versus the request path.
• Attempt to create objects that reference resources (like ConfigMaps or Secrets) in other namespaces.
• Test if controllers correctly validate the userInfo (username, groups) of the requester in their decision-making logic.

Hardening the Boundary: Defensive Controls

Testing reveals gaps; these controls close them.

1. Implement Zero-Trust Networking

Default-deny is the only sane starting point. Apply a baseline NetworkPolicy to every namespace that denies all ingress from other namespaces.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-cross-namespace
  namespace: <protected-namespace>
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {} # Only allow from pods in the SAME namespace

Explicitly allow required cross-namespace communication (e.g., from monitoring or service mesh namespaces) using namespaceSelector rules, not open CIDR blocks.

2. Principle of Least Privilege for Platform Tools

Scrutinize the RBAC for every cluster-scoped component. Does your admission controller really need create permission on pods cluster-wide? Or can its role be scoped to specific namespaces or resource types? Use tools like kubectl audit or third-party RBAC analyzers to find and reduce over-permissive bindings.

3. Namespace-as-a-Boundary for ServiceAccounts

Never bind a namespaced ServiceAccount to a ClusterRole unless it is absolutely necessary. If a component needs to act in multiple namespaces, create separate ServiceAccounts and RoleBindings in each namespace, or use a tightly scoped ClusterRole that only lists the required namespaced resources.

Monitoring for Boundary Violations

Detection is your last line of defense. Monitor the Kubernetes API audit logs for tell-tale signs of boundary crossing.

• Unauthorized Cross-Namespace API Calls: Alert on create, update, or get requests where the requesting user/service account’s namespace differs from the target object’s namespace, and the action is not explicitly allowed by a known pattern (e.g., cluster-admin, system components).
• Network Policy Violation Attempts: If using a CNI that supports it (like Cilium), export flow logs and alert on denied connection attempts from one namespace to a sensitive another.
• Admission Controller Anomalies: Monitor the logs of your admission webhooks for a high rate of denials or errors, which could indicate probing or exploit attempts.

Frequently Asked Questions

Conclusion: Isolation as an Active Discipline

Namespace isolation in Kubernetes is not a static configuration you set and forget. It is a dynamic security property that must be continuously validated, enforced, and monitored. The platform’s complexity and extensibility, while powerful, are the very factors that introduce fragility into this boundary.

The deprecation of a component like Ingress-NGINX teaches us to build frameworks for change. Similarly, breaches of namespace isolation teach us that security is not a feature of a single object (the namespace), but an emergent property of the entire system’s configuration. Your cluster’s security boundary is only as strong as the weakest link in the chain of RBAC, network policies, and admission control. Treat it as a critical, living part of your platform—one that demands proactive testing, rigorous hardening, and vigilant observation.