kubectl debug: The Complete Guide to Ephemeral Containers, Pod Copies and Node Debugging

kubectl exec stops working exactly when you need it most: the image has no shell, the container is stuck in CrashLoopBackOff, or the problem lives on the node rather than inside the pod. kubectl debug is the command built for those cases. It can attach a throwaway debug container to a pod that is already running, create a modified copy of a pod you can poke at safely, or drop you into a shell with the node’s filesystem mounted.

This guide covers all three modes, the --profile flag (whose default changed in kubectl 1.36), custom profiles, the RBAC and Pod Security rules that usually get in the way, and the limitations nobody mentions until you hit them, such as the fact that you cannot remove an ephemeral container once it is added. If your specific problem is a distroless image, the deep dive on debugging distroless containers goes further on that case; this article is the general reference for the command itself.

What kubectl debug Does (and Why exec Is Not Enough)

kubectl exec runs a new process inside an existing container. That has three hard requirements: the container must be running, the binary you want (usually sh) must exist in its image, and whatever tools you need must also be in that image. Modern production images break all three. Distroless and scratch images ship no shell, crashing containers are never running long enough to exec into, and nobody wants tcpdump or strace baked into an application image.

kubectl debug removes those requirements by bringing its own image. Depending on the target, it does one of three things:

The command is available in every supported kubectl version. Ephemeral containers, the API feature behind the first mode, have been stable since Kubernetes 1.25, so any cluster you are likely to run today supports them.

Mode 1: Ephemeral Debug Containers in a Running Pod

This is the mode people mean when they search for “kubectl debug container”. It attaches a new container to a pod that is already running, without restarting it:

kubectl debug -it mypod --image=busybox:1.36 --target=app

What happens under the hood: kubectl sends a PATCH to the pod’s ephemeralcontainers subresource, the kubelet starts the new container inside the same pod sandbox, and because of -it kubectl attaches your terminal to it. The ephemeral container automatically shares the pod’s network namespace, so localhost, the pod IP and the pod’s DNS configuration are exactly what the application sees. That alone makes it the fastest way to test connectivity from a pod’s point of view.

The –target flag and the process namespace

By default, containers in a pod do not see each other’s processes. The --target=<container> flag tells the runtime to put the debug container into the process namespace of the named container. With it, ps inside the debug container shows the application’s processes, and you can inspect them:

/ # ps aux
PID   USER     TIME  COMMAND
    1 65532     0:12 /app/server --port=8080
   23 root      0:00 sh
   29 root      0:00 ps aux

Two caveats from the official documentation. First, --target must be supported by the container runtime; containerd and CRI-O support it, but if it is not supported the ephemeral container may fail to start or start with an isolated process namespace, and ps will only show its own processes. Second, --target is only necessary when the pod does not already set shareProcessNamespace: true. If it does, every container, including the ephemeral one, already sees all processes.

Reading the application’s filesystem through /proc

Once you share the target’s process namespace, the application’s root filesystem is reachable through the proc filesystem, even though the debug container has its own image:

/ # ls /proc/1/root/app
config.yaml  server
/ # cat /proc/1/root/app/config.yaml

This is the trick that makes distroless images debuggable: you get your tools from busybox and the files from the target. If you get Permission denied on /proc/1/root, the debug container lacks the ptrace access it needs to look into another user’s process. Use --profile=general (which adds SYS_PTRACE) or, if you really need it, --profile=sysadmin. The profiles are covered in detail below.

Naming, detaching and reattaching

If you do not pass -c, kubectl generates a name like debugger-8xzrl. Name it yourself so you can find it again:

kubectl debug mypod -c debugger --image=nicolaka/netshoot --target=app -it

If your terminal disconnects, the container keeps running while its main process is alive. You can reattach with:

kubectl attach mypod -c debugger -it

When you type exit, the shell (the container’s main process) ends and the ephemeral container moves to Terminated. It is never restarted, and a terminated ephemeral container cannot be started again; you add a new one with a different name instead. Running kubectl describe pod mypod shows every ephemeral container ever added under an Ephemeral Containers: section.

Mounting volumes in a debug container

A common follow-up question is how to make kubectl debug mount a volume. The answer has two parts. You cannot add a new volume, because the pod spec’s volume list is immutable after creation. You can, however, mount a volume that the pod already declares, through a custom profile (see the profiles section) that sets volumeMounts:

# debug-mounts.yaml
volumeMounts:
- name: data
  mountPath: /data

kubectl debug mypod -it --image=busybox:1.36 --target=app \
  --profile=general --custom=debug-mounts.yaml

The name must match an entry in the pod’s spec.volumes. subPath mounts are not allowed for ephemeral containers. If what you need is just to read files the application sees, /proc/1/root is usually simpler than a mount.

Mode 2: Debugging a Copy of the Pod

Ephemeral containers are useless when the application container is crash-looping: there is no stable process to target, and you cannot change the application’s command. For those cases kubectl debug can create a copy of the pod with modifications, using --copy-to.

Changing the command of a crashing container

This is the standard fix for CrashLoopBackOff investigation. Create a copy where the failing container runs a shell instead of its entrypoint:

kubectl debug mypod -it --copy-to=mypod-debug --container=app -- sh

You now have a shell in a container with the same image, environment variables, volumes, and service account as the crashing one. From there you can inspect config files, check that mounted secrets exist, and run the original entrypoint by hand to watch it fail with full output.

The --container flag matters. The documentation is explicit: to change the command of an existing container you must name it, otherwise kubectl debug creates a new container to run the command you specified. This only works if the image has a shell; for distroless images, combine it with --set-image (next section) to swap in a debug variant.

Swapping images

--set-image changes images in the copy using the same container=image syntax as kubectl set image:

# Replace one container's image with a debug build
kubectl debug mypod -it --copy-to=mypod-debug --set-image=app=myregistry/app:1.4.2-debug

# Replace every container's image
kubectl debug mypod --copy-to=mypod-debug --set-image=*=ubuntu

Swapping to a :debug tag of the same application (many distroless-based images publish one with a busybox shell) is often the cleanest way to get a shell into an otherwise shell-less image.

Adding a debug container to the copy

You can also add a brand-new container to the copy, which works even when your cluster blocks ephemeral containers:

kubectl debug mypod -it --image=ubuntu --copy-to=mypod-debug --share-processes

--share-processes enables shareProcessNamespace in the copy so the debug container can see the application’s processes. It defaults to true when you use --copy-to, so you only need to spell it out for readability.

What gets copied and what does not

The copy is a standalone pod with no owner reference: it is not part of the original Deployment or ReplicaSet, so the controller does not manage it and it will not be replaced if it dies. Several fields are intentionally stripped so the copy behaves as a lab specimen and not as a production replica:

Two scheduling flags are worth knowing. --same-node schedules the copy on the same node as the original, which matters when you suspect a node-specific problem. --replace deletes the original pod after creating the copy; use it with care, since for a Deployment the controller will simply create a new replica.

Remember that the copy is a real pod consuming real resources. Delete it when you are done:

kubectl delete pod mypod-debug

Mode 3: Debugging a Node with kubectl debug node

When the problem is below the pod (kubelet errors, full disks, container runtime state, host networking), you can get a shell on a node without SSH:

kubectl debug node/worker-3 -it --image=ubuntu

kubectl creates a pod named like node-debugger-worker-3-pdx84 pinned to that node. According to the documentation, the container runs in the host’s IPC, network, and PID namespaces, and the node’s root filesystem is mounted at /host. That gives you access to host files such as /host/var/log and to host processes via ps.

The pod is not privileged by default, so reading some process information may fail and chroot /host may not work. If you need full access, use the sysadmin profile:

kubectl debug node/worker-3 -it --image=ubuntu --profile=sysadmin
# inside the container
chroot /host
crictl ps
journalctl -u kubelet --since "10 min ago"

This is particularly useful on immutable operating systems. On Talos Linux, for example, there is no SSH daemon at all, so node-level inspection goes through talosctl or a node debug pod; our Talos Linux guide covers that model.

Node debug pods are not cleaned up automatically. Delete them explicitly:

kubectl delete pod node-debugger-worker-3-pdx84

Debugging Profiles: –profile Explained

Every kubectl debug session applies a profile that decides the security context of the debug container. The profile matters because it determines what you can do (read other processes, capture packets, chroot to the host) and whether Pod Security admission accepts your debug container at all.

The descriptions follow the design in KEP-1441, which defines the profiles. The practical mapping is simple:

• Use general for everyday debugging; it is enough to read /proc/1/root and attach strace or a debugger to the application.
• Use netadmin with nicolaka/netshoot when you need tcpdump, iptables inspection or raw sockets.
• Use baseline or restricted when the namespace enforces those Pod Security Standards; other profiles will be rejected there.
• Use sysadmin only for node work or when nothing else works, and treat it like handing out root.

The default profile changed

For years, running kubectl debug without --profile meant legacy, and recent versions printed a warning about it. Following KEP-1441, general is the default from kubectl 1.36, and the legacy profile is scheduled for removal in 1.39. Note that the behavior depends on your kubectl client version, not on the cluster version. If you have scripts that rely on the old behavior, set --profile explicitly; it is good practice anyway because the profile is part of what you are asking the cluster to allow.

Custom profiles with –custom

Since Kubernetes 1.32, custom profiles are stable. A custom profile is a partial container spec in YAML or JSON that is merged on top of the static profile:

# custom-profile.yaml
env:
- name: JAVA_TOOL_OPTIONS
  value: "-XX:+UnlockDiagnosticVMOptions"
securityContext:
  capabilities:
    add:
    - NET_ADMIN
    - SYS_TIME

kubectl debug -it mypod --image=busybox:1.36 --target=app \
  --profile=general --custom=custom-profile.yaml

The documented constraints: a custom profile can only modify the container spec, not the pod spec, and it cannot change the container’s name, image, command, lifecycle or volumeDevices. Everything else, including env, securityContext, volumeMounts and workingDir, is fair game, which is how the volume mount example earlier works.

RBAC: What Permissions kubectl debug Needs

Most “kubectl debug does not work” reports are RBAC problems. The permissions differ by mode:

A minimal namespaced Role for ephemeral debugging looks like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ephemeral-debugger
  namespace: payments
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["patch", "update"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create"]

The error pods "mypod" is forbidden: User "dev" cannot patch resource "pods/ephemeralcontainers" means exactly the second rule is missing.

Be deliberate about who gets this. Patching pods/ephemeralcontainers lets a user run an arbitrary image inside any pod in the namespace, sharing its network and, with --target, its process space and filesystem. That is effectively the same trust level as pods/exec, and it should be granted and audited the same way. The Kubernetes security best practices article covers how to scope this kind of access, and namespace isolation is what keeps it contained.

Pod Security Admission and kubectl debug

Pod Security admission evaluates ephemeral containers too. In a namespace labeled pod-security.kubernetes.io/enforce=restricted, a debug container that runs as root or adds capabilities is rejected, and the error mentions the violated rule, for example allowPrivilegeEscalation != false or unrestricted capabilities. The fix is to pick the matching profile:

kubectl debug -it mypod --image=busybox:1.36 --target=app --profile=restricted

With restricted, the debug container runs as non-root with all capabilities dropped, so you lose SYS_PTRACE and most of /proc/1/root becomes unreadable unless the application runs as the same UID. That is a genuine limitation of debugging under the restricted standard, not a kubectl bug.

Node debugging has the same issue in a different place: the node debug pod uses host namespaces and a hostPath mount, which the baseline and restricted standards forbid. Since kubectl creates it in your current namespace, run node debugging from a namespace that allows it (typically a dedicated, privileged-labeled namespace such as node-debug) with -n.

Limitations You Should Know Before Using It in Production

You cannot remove an ephemeral container. Like regular containers, an ephemeral container cannot be changed or removed once added. There is no kubectl delete for it. When its process exits it stays in the pod’s status as Terminated until the pod itself is deleted. If a stack of terminated debug containers bothers you, the only cleanup is replacing the pod, for example with kubectl rollout restart deployment/<name>.

No resources, no ports, no probes. Ephemeral containers cannot declare resources, ports, or probes, because pod resource allocations and networking are fixed at creation. The debug container runs without its own requests, so a heavy tool (a JVM heap dump, a large tcpdump capture written to disk) competes with the application for the pod’s resources and node capacity. Keep sessions short and output small, and read resource requests and limits if you are unsure how much headroom the pod has.

No kubectl edit. Ephemeral containers are added through a dedicated API subresource, not by editing pod.spec, so you cannot add one with kubectl edit or kubectl apply.

Static pods are not supported. Ephemeral containers do not work on static pods, which includes control-plane components on kubeadm clusters. Use node debugging for those.

It only targets pods and nodes. kubectl debug accepts a pod, a node/<name>, or a file with -f. To debug a Deployment, pick one of its pods first, for example with kubectl get pods -l app=myapp.

The copy is not the original. A --copy-to pod runs on a fresh sandbox, has no live traffic, and may land on a different node. It reproduces startup and configuration problems very well and runtime or traffic-dependent problems poorly.

Choosing a Debug Image

The debug container can use any image your nodes can pull, so pick based on the job. Pin tags so your team gets the same toolset every time.

Practical Recipes

Debugging a CrashLoopBackOff pod

Start with the logs of the previous run, since they often contain the answer:

kubectl logs mypod -c app --previous

If the logs are empty or unhelpful, create a copy that does not start the app, then run it by hand:

kubectl debug mypod -it --copy-to=mypod-debug --container=app -- sh
/ # env | sort
/ # ls -la /etc/config
/ # /app/server --port=8080

For shell-less images, swap to a debug variant with --set-image, or add a busybox container with --target and read the files through /proc/1/root, the pattern covered step by step in debugging distroless containers.

Checking network connectivity from a pod

The ephemeral container shares the pod’s network namespace, so tests run from the pod’s real IP with its real DNS configuration and NetworkPolicies:

kubectl debug -it mypod --image=nicolaka/netshoot --profile=netadmin
~ # dig +search my-service
~ # curl -sv http://my-service.payments.svc:8080/healthz
~ # tcpdump -i any -nn port 5432

If the application sits behind a service mesh sidecar, remember that outbound traffic goes through the proxy; the article on Istio proxy DNS explains why DNS results can differ between the app and a plain debug container.

kubectl debug vs kubectl exec vs Pod Copy

The rule of thumb: use exec if the container is running and has what you need; use an ephemeral container when you need tools the image lacks while observing the live process; use a copy when the container cannot start or you need to change how it starts; use node debugging when the problem is not inside the pod. If you prefer a GUI, desktop clients like Freelens wrap exec and logs, but the three kubectl debug modes remain the most capable option.

Frequently Asked Questions

Conclusion

kubectl debug is three tools behind one command. The ephemeral container mode is what you reach for when a running pod misbehaves and its image has nothing useful in it; the copy mode is for containers that cannot start or need a different command or image; and the node mode replaces SSH for anything below the pod. Set --profile explicitly, grant pods/ephemeralcontainers as carefully as pods/exec, and remember that anything you attach to a live pod stays in its status until the pod is replaced. With those habits in place, a shell-less image stops being an obstacle to debugging.