You override a chart’s default livenessProbe with an exec command, deploy, and Kubernetes rejects it: “may not specify more than one handler type.” The chart’s default httpGet probe is still there, merged underneath your override, and now the pod spec has two probe handlers. You didn’t add it. You can’t see it in your values file. And the fix is a single character: ~.
That ~ is YAML’s null, and in Helm it does something most people never learn: setting a key to null deletes it from the merged values entirely, instead of setting it to a null value. It’s the cleanest way to remove a default a chart baked in — and it’s the tip of a whole set of null-handling behaviors (absent vs null vs empty, --set foo=null, default, required, hasKey) that quietly decide whether your templates render valid YAML or foo: <no value>.
This guide covers the null-deletes-a-key trick in depth, where it works and where it bites (the --reuse-values trap, the Helm 4 regression), and the related functions you need to tell “the user didn’t set this” apart from “the user set this to empty.” Everything is verified against current Helm docs and behavior.
First: ~ is just YAML null
Before Helm, this is pure YAML. All of these mean the same thing — null:
a: ~ # canonical shorthand
b: null
c: Null
d: NULL
e: # empty value is also null
~ is the canonical short form in the YAML spec; null/Null/NULL/empty are equivalent spellings. In a Helm values.yaml, writing foo: ~ is identical to foo: null. People reach for ~ because it’s terse and unmistakable — an empty value (foo:) is easy to misread as “I forgot to fill this in,” whereas foo: ~ reads as a deliberate null.
The Killer Trick: null Deletes a Default Key
Here is the behavior that makes ~ worth an article. When you override a chart’s values — with a -f values file, a parent chart overriding a subchart, or --set — Helm merges your values on top of the chart’s defaults. A normal override replaces a value. But a null override is special: Helm removes the key from the result.
Straight from the Helm docs (Chart Template Guide → Values Files):
“If you need to delete a key from the default values, you may override the value of the key to be null, in which case Helm will remove the key from the overridden values merge.”
The canonical example is the liveness-probe foot-gun from the intro. Say the chart defaults to:
You want an exec probe instead. If you just add your exec, the merge keeps the default httpGet — and a probe with both exec and httpGet is invalid. You delete the default with null:
# your override values.yaml
livenessProbe:
httpGet: ~ # ← deletes the chart's default httpGet
exec:
command: [cat, docroot/CHANGELOG.txt]
Or on the command line, exactly as the Helm docs show it:
The result has only the exec handler. Without the httpGet=null, both survive and Kubernetes rejects the manifest.
Why it works: Helm’s value coalescing walks the override tree onto the defaults. A present key with a real value overwrites; a present key with null is treated as an instruction to remove. This is the only way to subtract from a chart’s defaults — there’s no --unset flag (more on that below).
Where the null-delete works
The same mechanism applies anywhere Helm coalesces values:
A -f override file on top of the chart’s values.yaml (the docs example above).
A parent chart overriding a subchart. In the parent’s values.yaml, nulling a key under the subchart’s name deletes that subchart default:
“`yaml
# parent values.yaml
mysubchart:
someDefault: ~ # remove a default the subchart shipped
“`
--set key=null at install/upgrade time.
--set foo=null vs --set-string foo=null
On the CLI the distinction matters, and it’s a common source of “why didn’t it delete?”:
Command
Result
--set foo=null
foo becomes a real nil → key is deleted
--set-string foo=null
foo becomes the literal string"null" (no deletion)
--set a=null,name=[]
a: null, name: []
Helm’s --set parser does type conversion: the literal null (case-insensitive) is converted to a Go nil, which triggers the delete. --set-string forces everything to stay a string, so null is just the four-character word "null" — which is almost never what you want here. If your deletion isn’t happening, check you didn’t reach for --set-string.
Version Support (and the Helm 4 warning)
This is where “desde qué versión” gets interesting:
Helm 2 introduced deleting a key by setting it to null, but only reliably for top-level keys. Nested null-deletion (like web.livenessProbe.httpGet: null) was buggy — it could emit Cannot overwrite table item ... with non table value and override with null instead of deleting.
Helm 3 is where the documented behavior works as advertised, top-level and nested. If you’re on Helm 3, everything above is solid.
⚠️ Helm 4 — currently a regression. As of a still-open issue (filed March 2026), Helm 4 no longer reliably deletes keys via null: both foo: (blank) and foo: null can fail schema validation with errors like Invalid value: "null": ... must be of type string, especially against strict Kubernetes 1.34+ schemas. If you’ve moved to Helm 4, test null-deletion before relying on it — the behavior that was rock-solid in Helm 3 is in flux. This interacts directly with values JSON schema validation: a schema that types a field as string will reject a null, so schema and null-deletion can fight each other.
The --reuse-values Trap
The one place null-deletion does not do what you’d hope: helm upgrade --reuse-values.
--set foo=null deletes a key relative to the chart’s defaults. It does not cleanly “un-set” a value you previously set explicitly and are now carrying forward with --reuse-values — it overrides it with null rather than falling back to the chart default. There is a long-standing feature request for an explicit --unset flag precisely because this case has no clean answer today.
Practical rule: to genuinely reset a value back to the chart default on upgrade, prefer re-specifying your full intended values (-f) over leaning on --reuse-values plus --set x=null.
Absent vs null vs empty: The Trio That Trips Everyone
Deleting keys is half the story. The other half is reading optional values in templates — and Helm/Sprig blur three states that feel different: key absent, key present but null, and key present but empty (0, "", [], {}, false).
The critical thing to internalize: default, required, empty, and coalesce all treat nil, 0, "", empty list/map, and false as the same “empty.” They cannot tell “unset” from “set to zero.”
Bare-printing an unset value gives you the literal text <no value> in the manifest — which is not null, not empty, just a string that will confuse Kubernetes or your reader. (You may also see <nil> in some contexts; the exact literal is a Go-template detail that has shifted across Helm 3 minor versions, so don’t hard-code assumptions about which one appears — the point is it’s not what you want.)
The fixes:
Piping through toYaml turns nil into a proper null — use it for whole objects/maps: {{ .Values.config | toYaml | nindent 2 }}.
Piping through quote turns nil into "" — use it for optional scalars that must be strings.
Best of all, skip the key entirely when empty with with:
“`gotemplate
{{- with .Values.foo }}
foo: {{ . | quote }}
{{- end }}
“`
The with block is skipped for any empty/nil value, so an unset foo produces no line at all — the idiomatic Helm “omitempty.”
Switching Off a Subchart Block with null
One more practical use of ~. Because nil is “empty,” setting a value to null makes an {{ if }} guard fall through:
# subchart default turns something on
metrics:
serviceMonitor:
enabled: true
```
```yaml
# parent override switches it off cleanly
metrics:
serviceMonitor: ~ # or: enabled: false
Nulling the whole block (or the flag) makes {{ if .Values.metrics.serviceMonitor }} evaluate false — a tidy way for a parent chart or an environment override to disable a section a subchart enabled by default, using the same empty-semantics as everything above.
Cheat Sheet
Goal
Syntax
Write a null in values
foo: ~ (or foo: null)
Delete a chart’s default key
override it with ~ / null
Delete via CLI
--set foo=null (not--set-string)
Fallback for empty/unset
`{{ .Values.foo \
default “x” }}`
Distinguish null from absent
{{ if hasKey .Values "foo" }}
Require a value
{{ required "msg" .Values.foo }}
Nested optional read
{{ dig "a" "b" "fallback" .Values }}
Null-safe object into YAML
`{{ .Values.obj \
toYaml \
nindent 2 }}`
Omit a key when empty
{{- with .Values.foo }} … {{- end }}
Wrapping Up
~ in Helm is a two-job character: it writes a YAML null, and — the part almost nobody documents in their own charts — it deletes a default key from the merged values, which is the only clean way to subtract a probe, an annotation, or a whole block that a chart shipped by default. Around it sits a set of null rules worth memorizing: default/required/empty can’t tell unset from zero (use hasKey when that matters), and an unset value bare-printed becomes <no value> unless you route it through toYaml, quote, or with.
Two warnings to carry: --reuse-values doesn’t cleanly un-set values, and Helm 4’s null-deletion is currently a regression — so if you’re on 4.x, verify before you rely on it.
GPU scheduling in Kubernetes used to be deceptively simple: install the NVIDIA device plugin, request nvidia.com/gpu: 1, and let the default scheduler find a node with one available GPU. That model got many clusters into production, but it encoded the wrong abstraction. A modern GPU is not just an integer. It has memory size, architecture, interconnect, topology, partitioning modes, sharing modes, and health state.
Dynamic Resource Allocation (DRA) is Kubernetes’ answer to that mismatch. The core DRA APIs graduated to GA in Kubernetes 1.34, with the stable resource.k8s.io/v1 API enabled by default. In 2026, this matters because the ecosystem around AI infrastructure has also moved: NVIDIA donated its DRA Driver for GPUs to the Kubernetes community under CNCF governance at KubeCon Europe 2026, Kueue has native concepts for DRA-aware quota management, KAI Scheduler is a CNCF Sandbox project for large GPU fleets, and inference stacks such as vLLM and KServe are becoming the runtime layer above the scheduler.
This is not a “replace one YAML key with another” migration. DRA changes where device knowledge lives. Instead of asking Kubernetes for a count of opaque extended resources, workloads request a claim against a class of devices, and the scheduler allocates a concrete device that satisfies the claim.
Why the device-plugin model is reaching its limits
The Kubernetes device plugin framework exposes vendor devices to the kubelet. For NVIDIA GPUs, the traditional resource name is nvidia.com/gpu, requested in container resources.requests and resources.limits. This remains useful, especially for simple clusters.
The limitation is in the API shape. Kubernetes extended resources are integer resources and cannot be overcommitted. The Kubernetes documentation also states that devices cannot be shared between containers through the basic extended-resource model. That is fine for “one pod owns one whole GPU”. It is much weaker for LLM inference, mixed training queues, fractional capacity, MIG profiles, topology-sensitive multi-GPU jobs, and heterogeneous node pools.
The device-plugin model also pushes too much meaning into out-of-band policy. If you need A100s rather than L4s, you usually add node labels, node affinity, taints, or separate node groups. If you need a MIG slice, you configure GPU Operator and device plugin strategy, then expose separate resource names or labels. If you need low-latency multi-GPU placement, you combine scheduler plugins, topology labels, and workload-specific conventions.
Those workarounds fragment the source of truth. The scheduler sees integer capacity. The driver knows device details. The autoscaler knows node templates. The ML platform knows model requirements. DRA gives Kubernetes a structured device model so those systems can coordinate through API objects.
For node provisioning, this does not remove the need for good autoscaling. You still need a node autoscaler that can bring up GPU capacity with the right labels, taints, AMI, driver stack, and instance family. If you run on AWS, /cluster-autoscaler-vs-karpenter/ is directly relevant: GPU pods sitting Pending are often a node provisioning problem before they are a scheduler problem. On EKS, /eks-auto-mode/ is also worth reading because managed node lifecycle and accelerator support change how much of the stack you own.
What DRA actually adds
DRA introduces Kubernetes APIs for claiming devices. The stable API group is resource.k8s.io/v1. The important objects are:
API object
Scope
Purpose
DeviceClass
cluster
Defines a category of devices and optional selectors/configuration. Claims reference a DeviceClass.
ResourceSlice
cluster
Published by DRA drivers. Describes available devices, attributes, capacity, and node access.
ResourceClaim
namespace
Requests access to devices. The scheduler allocates concrete devices into the claim status.
ResourceClaimTemplate
namespace
Template for per-pod ResourceClaims, similar in spirit to volume claim templates.
Pod spec.resourceClaims
pod
Makes a ResourceClaim or ResourceClaimTemplate available to the pod.
Container resources.claims
container
Attaches a named claim to a specific container.
The control flow is simple. A DRA driver publishes devices as ResourceSlice objects. A cluster administrator or driver provides DeviceClass objects. A workload creates a ResourceClaim or references a ResourceClaimTemplate. During scheduling, Kubernetes evaluates the claim against ResourceSlices, picks devices on nodes where the pod can run, stores the result in ResourceClaim status, and the driver prepares the device for the pod.
This is close to the PersistentVolumeClaim mental model: a pod references a claim with requirements, and Kubernetes binds it to something concrete. With DRA, the pod claims a device from a class, with selectors and constraints that drivers and the scheduler can reason about.
Do not overstate this: DRA is not live GPU hot-plugging, and it is not a model-serving platform. It is a scheduling-time allocation framework. You still need CPU and memory requests to be correct, as covered in /kubernetes-resource-requests-limits/. You still need queueing for batch fairness and an inference runtime for serving.
A minimal DRA GPU request
The exact DeviceClass names and available attributes depend on the installed driver. NVIDIA’s NIM Operator documentation says the NVIDIA DRA driver deploys a default gpu.nvidia.com DeviceClass for physical GPUs. Always verify this in your cluster:
kubectl get deviceclasses
kubectl get resourceslices
For a single GPU per pod, use a ResourceClaimTemplate:
This manifest uses the stable resource.k8s.io/v1 API and the pod-level resourceClaims plus container-level resources.claims fields shown in the Kubernetes DRA task documentation. It deliberately does not use the legacy nvidia.com/gpu resource request.
In production, you will usually add selectors. Kubernetes supports CEL selectors in DRA claims, but attribute names are driver-specific. Inspect ResourceSlices before standardizing selectors:
kubectl get resourceslices -o yaml
For example, a platform team might publish DeviceClasses such as “inference-l4”, “training-h100”, or “mig-1g-10gb” instead of asking application teams to write low-level CEL expressions.
MIG, sharing, and topology awareness
MIG is where DRA becomes more than nicer syntax. With the device-plugin model, MIG support works, but the cluster often ends up with a mixture of resource names, node labels, and operational conventions. DRA lets the driver publish device shapes and capacities in ResourceSlices.
NVIDIA’s current DRA driver documentation is cautious: the driver manages GPUs and ComputeDomains; ComputeDomains are officially supported for robust and secure Multi-Node NVLink, while some GPU allocation features are still described as exploratory in the upstream README. NVIDIA’s NIM Operator documentation already covers Kubernetes v1.34 or later, resource.k8s.io/v1, the gpu.nvidia.com DeviceClass, full GPU versus MIG decisions, and the GPU Operator path. Pin driver versions and test the exact mode you intend to offer.
Topology matters at two levels.
First, there is node-local topology: NUMA locality, PCIe lanes, NVLink, and whether devices are close to the CPUs and memory the pod uses. Kubernetes has long had a Topology Manager in kubelet, but DRA gives the scheduler more structured information before placement.
Second, there is fleet topology: racks, blocks, zones, and inter-node fabric. Kueue’s Topology Aware Scheduling documentation targets AI/ML workloads where pod-to-pod bandwidth affects runtime and cost. DRA handles device allocation; Kueue decides when a workload should be admitted and how scarce quota should be shared.
Preemption belongs in that same layered view. Kubernetes scheduler preemption and Kueue preemption can make room for higher-priority workloads, but DRA itself is the device allocation substrate. In practice, you combine DRA claims, PriorityClasses, Kueue ClusterQueues, and possibly KAI Scheduler policies to get predictable multi-tenant behavior.
What NVIDIA’s CNCF donation changes
On March 24, 2026, NVIDIA announced at KubeCon Europe in Amsterdam that it was donating the NVIDIA DRA Driver for GPUs to CNCF, moving it from vendor governance to community ownership under the Kubernetes project.
It changes the risk profile for platform teams. A GPU DRA driver under Kubernetes community governance is easier to treat as part of the cloud-native substrate, and it creates a clearer collaboration point for cloud providers, Kubernetes SIGs, Kueue, KAI Scheduler, and inference platforms.
It does not make NVIDIA-specific hardware vendor-neutral. CUDA, MIG, NVLink, GPU Operator, and driver lifecycle remain NVIDIA concerns. What improves is the Kubernetes integration surface for advertising, claiming, allocating, and preparing devices.
NVIDIA also announced that KAI Scheduler had been onboarded as a CNCF Sandbox project. CNCF describes KAI Scheduler as a Kubernetes scheduler for optimizing GPU resource allocation for AI workloads in large-scale clusters. DRA models and allocates devices; KAI provides AI-focused scheduling policy; Kueue provides quota, admission, fair sharing, and preemption; KServe and vLLM provide the inference serving layer.
For production inference, vLLM and KServe are consumers of GPU scheduling, not replacements for it. vLLM documents KServe integration for distributed model serving, and KServe documents multi-node, multi-GPU inference using a vLLM serving runtime.
Migration: device plugin to DRA
Do not migrate the whole fleet in one step. A practical migration looks like this:
Inventory existing GPU workloads. Classify them by device shape: whole GPU training, small inference, MIG-friendly inference, multi-GPU single-node, and multi-node training.
Upgrade the control plane and nodes to Kubernetes 1.34 or later. Verify resource.k8s.io/v1 is available with kubectl api-resources | grep resource.k8s.io.
Install or upgrade the GPU Operator and NVIDIA DRA driver in a small, isolated GPU node pool.
Verify DeviceClass and ResourceSlice objects before writing application selectors.
Define platform-owned DeviceClasses for common use cases. Prefer “h100-training”, “l4-inference”, or “mig-small” over asking every team to understand device internals.
Convert one non-critical workload from nvidia.com/gpu to a ResourceClaimTemplate. Keep CPU and memory requests unchanged unless you are intentionally resizing the workload.
Add Kueue for batch admission if multiple teams compete for GPUs. Use ClusterQueues, ResourceFlavors, cohorts, fair sharing, and preemption policies.
Validate node provisioning. If the claim is valid but no node exists, the autoscaler still has to create the right GPU node.
Roll into serving platforms. For KServe/vLLM, verify how the serving controller passes or creates DRA claims.
Retire legacy paths only after observability, rollback, and quota policies are in place.
During migration, it is reasonable to run legacy device-plugin workloads and DRA workloads side by side on separate node pools. Avoid advertising the same physical GPU through two allocation systems to the same scheduling domain unless the driver documentation explicitly supports that configuration.
Device plugin vs DRA
Capability
Device plugin / extended resource
DRA
Workload request
nvidia.com/gpu: 1 integer resource
ResourceClaim or ResourceClaimTemplate
API maturity
Device plugin framework exists since Kubernetes 1.10 beta
Core DRA APIs GA in Kubernetes 1.34
Device attributes
Mostly external labels and conventions
Structured attributes/capacity through ResourceSlices
Sharing
Limited by extended-resource model; vendor-specific sharing modes
Claims can model sharing patterns when supported by driver and feature set
MIG / partitions
Works through vendor configuration and resource exposure
Better fit for requesting specific device shapes
Scheduler awareness
Counts resources on nodes
Allocates concrete devices during scheduling
Autoscaler visibility
Often depends on node templates and resource names
Structured parameters improve simulation potential, but autoscaler support still matters
Best fit
Simple whole-GPU workloads
Heterogeneous, partitioned, shared, or topology-sensitive GPU fleets
When you do not need DRA yet
DRA is not mandatory for every GPU cluster in 2026.
If every workload needs exactly one whole GPU on a homogeneous node pool, the device plugin model may be simpler. If your managed Kubernetes provider does not support the driver path you need, waiting is rational. If your bottleneck is cold node provisioning, image pull time, model download time, or bad CPU/memory requests, DRA will not fix that by itself.
Pilot DRA when multiple teams compete for expensive GPUs, you run mixed GPU models, MIG or sharing is a first-class requirement, multi-GPU topology affects performance, or you need cleaner integration between scheduling, quota, and AI workload platforms.
FAQ
Is DRA GA in Kubernetes?
The core DRA APIs graduated to GA in Kubernetes 1.34, using resource.k8s.io/v1 and enabled by default. Some surrounding features continued to mature later; for example, Kubernetes documentation in 2026 marks certain DRA task flows as v1.35 [stable] and DRA prioritized lists as v1.36 [stable].
Does DRA replace the NVIDIA device plugin?
For DRA workloads, the claim replaces the nvidia.com/gpu extended-resource request. Operationally, many clusters will run both models during migration. The NVIDIA DRA driver is the relevant component for DRA-based GPU allocation.
Can I request fractional GPUs with DRA?
DRA provides a framework for richer requests and sharing, but the exact behavior depends on Kubernetes feature maturity and the driver. NVIDIA documents MIG and time-slicing paths in its GPU stack, while DRA consumable capacity adds a Kubernetes model for sharing device capacity. Validate the exact mode you want before promising fractional GPU self-service.
Is DRA enough for multi-tenant GPU scheduling?
No. DRA allocates devices. For fair sharing, quota, admission, and preemption across teams, add Kueue or a scheduler layer such as KAI Scheduler, depending on your workload shape.
Does DRA help with inference?
Yes, but indirectly. DRA makes the GPU request more accurate. vLLM and KServe still handle serving concerns such as model runtime, scaling, routing, and multi-node inference patterns.
CTA: pilot DRA in one GPU node pool
The right first step is one GPU node pool. Install the NVIDIA DRA driver, verify DeviceClass and ResourceSlice objects, and convert one smoke-test Job plus one real low-risk workload to ResourceClaimTemplate. Measure scheduling latency, claim allocation status, GPU utilization, failure modes, and autoscaler behavior. If the pilot is boring, expand it to one tenant queue. If it is noisy, fix the platform contract first.
Sources
Kubernetes v1.34 DRA GA announcement: https://kubernetes.io/blog/2025/09/01/kubernetes-v1-34-dra-updates/
Kubernetes node operating systems have become a real design choice again. For years, most teams ran Ubuntu, Debian, RHEL, or Amazon Linux on workers, then layered kubelet, containerd, hardening, patching, observability agents, GPU drivers, and configuration management on top. That works, but it leaves a large mutable host underneath a scheduler that treats nodes as replaceable capacity.
Immutable container-focused operating systems attack the problem differently. They reduce the host to a small, versioned artifact, make updates atomic, and move configuration into declarative APIs or boot-time documents. The goal is fewer moving parts, less drift, smaller attack surface, predictable upgrades, and replaceable nodes.
In 2026, the three names that come up most often are Talos Linux, Bottlerocket, and Flatcar Container Linux. They overlap, but they are not interchangeable:
Talos Linux is the most Kubernetes-specific and the most radical: no SSH, no shell, no package manager, API-only operations.
Bottlerocket is the AWS-native container host: strong fit for EKS and ECS, image variants, API settings, admin containers, and AWS support.
Flatcar Container Linux is the evolutionary successor to the CoreOS Container Linux model: read-only system partition, Ignition/Container Linux Config, systemd, SSH, automatic updates, and broad platform coverage.
For a step-by-step Talos implementation guide, read the deep dive: Talos Linux: The Immutable, API-Driven OS for Kubernetes. This article is a decision guide: what changes operationally, where each OS fits, and what trade-offs senior platform teams should care about.
The decision in one paragraph
Choose Bottlerocket when your platform is AWS-based and you want a supported node OS that integrates with EKS, EC2, ECS, AMIs, and AWS support plans. Choose Talos when you want a Kubernetes-first OS across bare metal, edge, private cloud, or multi-cloud, and can replace SSH-era habits with an API-driven model. Choose Flatcar when you want immutable infrastructure and automatic updates, but still need systemd, SSH, Ignition-style provisioning, and a smoother migration from Ubuntu or CoreOS-like patterns.
What Talos Linux is
Talos Linux is a Kubernetes-optimized Linux distribution from Sidero Labs. Its design is deliberately narrow: it exists to run Kubernetes. It is not a general-purpose server distribution with a hardened profile. The host has no SSH daemon, no interactive shell, and no package manager. Configuration and day-2 operations go through talosctl and an authenticated gRPC API using mutual TLS.
That changes the security model. On a traditional node, operational access usually means SSH, sudo, shell history, mutable filesystems, ad hoc commands, and sometimes a long tail of packages that were installed to debug one incident and never removed. Talos removes that path. You inspect resources, logs, services, disks, and Kubernetes component state through the Talos API. You change machine configuration by applying declarative config patches. You upgrade by asking the node to move to a new installer image.
The upside is a very small node surface. Sidero states that Talos ships with fewer than 50 OS binaries and no package manager; current releases also publish SBOM, VEX, and cryptographic signatures. Talos upgrades use an A/B image scheme so the previous kernel and OS image remain available for rollback. As of this review, the latest GitHub release is Talos v1.13.5, published on June 22, 2026.
The cost is operational discipline. If your incident response muscle memory is ssh, strace, edit a file, restart a daemon, Talos forces a redesign around API resources, Kubernetes-level observability, ephemeral debug containers where appropriate, and reproducible configuration.
What Bottlerocket is
Bottlerocket is an open-source Linux-based operating system purpose-built by Amazon Web Services for running containers. AWS provides Bottlerocket as no-cost AMIs for EC2 and documents it as a container host for EKS, ECS, VMware, and bare metal. The product center of gravity is clearly AWS.
Bottlerocket does not use a general-purpose package manager. Instead, it ships as variants: predefined images tailored to platform, architecture, orchestrator, orchestrator version, and sometimes flavor. Kubernetes variants are tied to Kubernetes minor versions, and Bottlerocket also has NVIDIA and FIPS flavors. The Bottlerocket OS version is separate from the Kubernetes variant version. As of this review, the current docs are under the 1.62.x branch and the latest GitHub release is v1.62.1, published on June 22, 2026.
Configuration is modeled and exposed through an API. Updates are image-based and use partition flips rather than package mutation. For Kubernetes clusters, the Bottlerocket Update Operator, or Brupop, coordinates in-place node updates. It runs a controller, node agents, and an API server, drains nodes through the eviction API, respects PodDisruptionBudgets, and rolls updates in waves.
Bottlerocket is less austere than Talos. It supports host and admin container concepts for controlled access, which helps with break-glass paths, SSM-style workflows, and AWS support expectations. The trade-off is portability: it can run outside EC2, but its strongest ecosystem, release consumption model, and support story are AWS-shaped.
What Flatcar Container Linux is
Flatcar Container Linux is a community Linux distribution for container workloads. It started as a fork of CoreOS Container Linux and now lives in the Cloud Native Computing Foundation. CNCF lists Flatcar as an Incubating project, accepted on August 2, 2024.
Flatcar is immutable in the classic Container Linux sense. The system partition is read-only, the OS is delivered as an image, updates are automatic and atomic, and the node is provisioned declaratively at boot. Unlike Talos, Flatcar keeps a familiar Linux model: systemd, SSH, files, units, and host-level customization. The current recommended provisioning path is Container Linux Config transpiled to Ignition. The older cloud-config model remains useful historically, but the docs recommend Container Linux Config for new provisioning.
Flatcar’s platform coverage is broad, including major clouds, OpenStack, Proxmox, VMware, KubeVirt, libvirt, QEMU, and VirtualBox. The release page currently shows Stable 4593.2.3, Beta 4694.1.0, Alpha 4722.0.0, and LTS 4081.3.8. The LTS model is useful for teams that want fewer feature changes.
Flatcar’s day-2 profile is closest to traditional Linux, which is both its strength and its risk. You can carry forward systemd-based automation, SSH debugging, and existing tools. You can also carry forward drift if you treat it like a mutable server.
Comparative table
Area
Talos Linux
Bottlerocket
Flatcar Container Linux
Core philosophy
Kubernetes-only, API-managed, no shell
AWS-built container host with variants
Container Linux successor with systemd and Ignition
Best fit
Bare metal, edge, private cloud, multi-cloud, high-security clusters
EKS/ECS and AWS-heavy fleets
Ubuntu/CoreOS-style migrations that still need Linux primitives
Management model
talosctl over mTLS gRPC; declarative machine config
API settings, variants, AMIs, Brupop
Container Linux Config/Ignition, systemd, SSH, channels, Nebraska
SSH and shell
No SSH, no shell
Not a normal SSH-first OS; controlled access through host/admin containers
SSH and normal Linux debugging patterns are available
Package manager
None
None; use variants instead
No traditional package workflow for the base OS; extend through containers, Ignition, systemd, and systemd-sysext
Open-source Talos by Sidero Labs; Sidero is a CNCF Silver Member
Open-source project built by AWS; AWS supports AWS-provided builds
CNCF Incubating project since August 2024
Learning curve
Highest if the team expects SSH
Moderate for AWS teams; lower inside EKS practices
Lowest for Linux operations teams
Day-2 operations: upgrades, debugging, and extensions
The day-2 question matters more than installation. You will install the OS once per node lifecycle, but upgrade, debug, replace, patch, and explain incidents for years.
Talos is the most coherent if you want every node operation to be declarative and auditable. Upgrades are API calls against an installer image. Kubernetes and OS upgrades are explicit operations, not package side effects. Diagnostics come from talosctl: service status, logs, disks, network state, component health, and resource APIs. Runbooks need Talos resources, not shell commands.
Extensions in Talos are powerful but constrained. System extensions are included in generated boot media or installer images, commonly via Image Factory. Extension services can run early privileged services where static pods or DaemonSets are not enough. Hardware-specific work, such as NVIDIA support, is possible but release-bound.
Bottlerocket day-2 operations fit AWS lifecycle thinking. In EKS, Brupop handles in-place updates for existing Bottlerocket nodes. It drains nodes, uses the eviction API, respects PDBs, and limits rollout blast radius. New nodes still depend on the AMI or image you launch, so keep launch templates current.
Debugging Bottlerocket is less disruptive than Talos for teams that want controlled host access, because the OS supports admin and host containers. Treat that as break-glass access, not routine configuration.
Flatcar day-2 operations are the most familiar. Automatic updates arrive through channels. Production clusters usually use Stable; a subset can run Beta to catch issues earlier. Reboot behavior is controlled through documented strategies, and organizations can run their own Nebraska-style update infrastructure. Flatcar also has an LTS channel.
For extensions, Flatcar has become more modular through systemd-sysext: overlays on read-only /usr, prebuilt sysext-bakery images, Kubernetes binaries, CRI-O, K3s, Wasmtime, and cloud vendor tooling.
When to choose Talos
Choose Talos when Kubernetes is the product boundary of the node. If nothing should run on the host except Kubernetes machinery, Talos gives you the cleanest model. It is compelling for bare metal, edge clusters, private cloud, Proxmox/vSphere, and multi-cloud fleets where you want one operational abstraction.
Talos is also the strongest choice when security teams want a hard answer to SSH access, mutable root filesystems, package drift, and undocumented host changes. Pair it with Kubernetes security best practices and the node layer becomes easier to audit.
Do not choose Talos casually if your team depends on node-level manual debugging, package-installed host agents, or mixed workloads outside Kubernetes. Talos rewards teams that invest in runbooks, observability, automated replacement, and reproducible node configuration.
When to choose Bottlerocket
Choose Bottlerocket when most of your Kubernetes fleet runs on AWS, especially EKS, and your organization values AWS-native support paths. Bottlerocket aligns with EC2 images, EKS worker patterns, AWS-provided builds, and AWS support plans.
Bottlerocket also fits teams that want immutable nodes but cannot go all the way to Talos’s no-shell model. Admin and host containers provide controlled access, variants reduce ambiguity, and Brupop coordinates rollout.
The caveat is portability. Bottlerocket can run in non-AWS contexts, but the further you move from AWS, the more you should compare it against Talos and Flatcar on ecosystem, tooling, and support.
When to choose Flatcar
Choose Flatcar when you want a lower-risk transition from traditional Linux nodes to immutable infrastructure. It is strong for teams that already understand systemd, Ignition, CoreOS-style provisioning, SSH, update channels, and cloud images. It gives you a read-only base OS, automatic atomic updates, broad platform availability, and CNCF governance without rewriting every host-level assumption.
Flatcar is also useful when Kubernetes is not the only container workload, or when Kubernetes, standalone container hosts, and custom systemd-managed components coexist. It is less opinionated than Talos and less AWS-centered than Bottlerocket.
The risk is familiarity. If engineers SSH into nodes, add one-off files, and treat systemd units as snowflake configuration, the immutable base loses value. Manage Flatcar with boot-time config, versioned artifacts, controlled update channels, and replaceable nodes.
FAQ
Is Talos more secure than Bottlerocket or Flatcar?
It depends on what you mean by secure. Talos has the smallest operational attack surface because it removes SSH, shell access, and package management entirely. Bottlerocket and Flatcar also reduce attack surface compared with general-purpose distributions, but preserve different forms of host access and customization.
Can Bottlerocket run outside AWS?
Yes, but AWS remains its center of gravity. Official materials describe variants for Kubernetes worker nodes in EC2, VMware, and bare metal, and ECS variants for EC2. For non-AWS-first environments, compare platform support, image lifecycle, and support expectations against Talos and Flatcar.
Is Flatcar just old CoreOS?
No. Flatcar started as a fork of CoreOS Container Linux, but it is now an active CNCF Incubating project with its own release streams, platform support, update mechanisms, and systemd-sysext-based modularity.
Which one is easiest to migrate to from Ubuntu worker nodes?
Flatcar is usually easiest because it preserves systemd, SSH, and a recognizable Linux host model while adding immutable OS updates. Bottlerocket can be straightforward inside EKS. Talos is often the biggest shift because it changes how administrators access and debug nodes.
Final recommendation
For AWS-heavy EKS fleets, start with Bottlerocket unless portability or a no-shell requirement points elsewhere. For bare metal, edge, private cloud, and multi-cloud Kubernetes, put Talos at the top of the shortlist and validate debugging and extension workflows early. For teams modernizing from Ubuntu or legacy CoreOS-style infrastructure, Flatcar is the pragmatic bridge.
The bigger decision is whether your platform treats nodes as replaceable, versioned infrastructure. Once that is true, all three can work well. The best choice is the one whose failure modes match your team’s reality.
Log management and observability are no longer a matter of “storing text and searching it quickly.” In 2026, the real challenge is economic and architectural: ingesting growing volumes, retaining data for months, correlating logs with metrics and traces, and responding during an incident without costs spiraling out of control.
For years, Elasticsearch was the dominant option. Then came OpenSearch, Loki, Quickwit and, with significant momentum, ClickHouse as an analytical backend. All of them can solve part of the problem, but they do not optimize for the same things. The senior decision should not be “which one is faster” in the abstract, but which questions the team will ask, what retention it needs and how much complexity it can operate.
The Historical Standard: Elasticsearch and OpenSearch
OpenSearch began as an open source fork of Elasticsearch and keeps the mental model most familiar to many teams: indices, shards, replicas, mappings, Query DSL, aggregations and a mature full-text experience. If your organization already comes from Elasticsearch, it is the most direct transition.
Architecturally, it is built on Apache Lucene. Each index is divided into shards; each shard maintains segments and inverted indices; the cluster coordinates writes, replicas, distributed searches and aggregations. This provides a great deal of power, but also a clear operational footprint: JVM, heap sizing, merges, hot/warm/cold tiers and mapping tuning.
For logs, OpenSearch shines when you need expressive text search, rich filters, aggregations, mature dashboards, existing tooling or use cases close to SIEM. The cost appears as volume grows: indexing a lot of content makes it fast to search, but consumes storage and memory.
In 2026, the 3.x branch also matters. OpenSearch 3.0 introduced relevant changes such as Lucene 10 and JVM 21, along with performance improvements and vector search capabilities. That signals project vitality, but it also means upgrades need more careful planning than in a purely stateless stack.
Loki: Observability Centered on Efficiency
Grafana Loki takes a different philosophy: do not index the full content of logs, only the labels. In practice, it is closer to “Prometheus for logs” than to Elasticsearch: queries by labels and subsequent filtering over lines with LogQL.
Its storage is based on compressed chunks and a small index. Chunks usually live in object storage such as S3, GCS, Azure Blob or MinIO, while the index points to streams defined by label combinations. That decision reduces cost, but shifts responsibility to label design.
The practical rule is simple: labels must have controlled cardinality. cluster, namespace, app, environment or service usually work well. request_id, user_id or trace_id as labels can break the model because they multiply streams, hurt compaction and grow the index. For those values, it is better to filter log content or rely on traces.
Loki shines in Kubernetes because the label model fits pods, namespaces and services. Its Grafana integration is also excellent: log exploration, jumps from metrics, alerts with LogQL and correlation with Tempo or Prometheus.
Its weakness appears when you want to treat logs as an arbitrary search corpus. A query like “find this fragment across all logs from the last six months” can be much more expensive than in a system with a full-text index. Loki is designed for queries bounded by labels and time, not to replace a general-purpose search engine.
In 2026, Loki remains highly active. The official documentation lists recent 3.x versions, including the 3.7 branch, and Grafana continues to position it as a horizontally scalable, highly available and cost-efficient log aggregation system. Grafana’s acquisition of Logline further reinforces the focus on difficult searches within logs, although it does not change the central principle: label design remains critical.
When Loki Shines
Loki is the pragmatic choice for Kubernetes teams that already live in Grafana, need to retain volume at low cost and query by service, namespace, cluster, environment and time window. It is less convincing for global full-text search or deep forensic analysis.
Quickwit: Cloud-Native Search on Object Storage
Quickwit is a distributed search platform written in Rust, designed for append-only workloads such as logs and traces. Its thesis is attractive: keep powerful search, but decouple compute and storage by using object storage as the main layer.
Instead of depending on hot local disks, Quickwit writes indices and splits to S3-compatible storage, Azure Blob, Google Cloud Storage or MinIO. Indexers process data, searchers query segments and the metastore coordinates state. The model uses inverted indices optimized for immutable data and remote storage: closer to OpenSearch in search than Loki, but with cloud-native economics.
Quickwit is especially interesting for teams that want to replace part of an expensive Elasticsearch/OpenSearch cluster, already have logs in S3 or can move them there, and need text search with a better operational cost profile. It also has Grafana integration and APIs that are partially compatible with Elasticsearch/OpenSearch, which helps with some existing workflows.
The critical nuance for 2026 is its project status. Datadog acquired Quickwit in 2024. The acquisition did not mean the open source repository disappeared: the quickwit-oss/quickwit repository remains public and shows recent activity, including 2026 tags and discussions. The Quickwit Grafana plugin also publishes recent compatibility with Grafana 12.1+ and 13. That said, the product risk is different from Loki or OpenSearch: strategic direction is influenced by Datadog, and part of the effort may be oriented toward internal or commercial use cases.
The honest reading is this: Quickwit should not be dismissed as “dead,” because according to the reviewed sources it is not archived. But it should be evaluated with more diligence: release cadence, critical issues, roadmap, real API compatibility, operational ease and dependency on the maintainer team. For a core observability platform, that evaluation matters as much as the benchmark.
ClickHouse: The Analytical Engine of New Observability
ClickHouse was born as a columnar analytical database, not as a log platform. That is exactly why it has become relevant to observability. Many incidents are not solved by searching for an exact string, but by grouping by service, calculating percentiles, exploring cardinalities, joining logs with traces and querying billions of events over a time range.
ClickHouse’s model is columnar. Data is stored by columns, compresses extremely well and is queried with vectorized execution, ordered primary indices, data skipping indexes and engines such as MergeTree. For observability, this makes it possible to store wide events and read only the columns needed.
The tradeoff is that ClickHouse is not Elasticsearch. Full-text search exists and has improved, but its strength remains structured and semi-structured analysis. If your queries are “give me all 500 errors by version, region and endpoint in the last 15 minutes,” ClickHouse is a very good fit. If your main query is “find any line containing a rare phrase across six months of unstructured text,” you probably want another component or a hybrid strategy.
ClickHouse has moved decisively toward observability. The acquisition of HyperDX and the launch of ClickStack change its positioning: it is no longer only “use ClickHouse as the backend and build the UI yourself,” but a stack with OpenTelemetry, HyperDX as the interface and ClickHouse as the engine. It shines in high-volume platforms, structured logs, traces, derived metrics and low-latency aggregate queries, although it requires understanding modeling, TTLs, compression and ingestion costs.
Quick Comparison
Characteristic
OpenSearch
Loki
Quickwit
ClickHouse
Main model
Distributed search on Lucene
Logs by streams and labels
Cloud-native distributed search
Columnar analytical database
Storage model
Shards, Lucene segments, hot/warm/cold tiers
Compressed chunks + small index
Indices/splits in object storage
Compressed columns in MergeTree
Index
Inverted, very complete
Labels, not full content
Inverted, optimized for object storage
Primary index, data skipping, columns
Full-text search
Excellent
Limited and dependent on filters
Very good
Good, not its main strength
Relative cost
High
Low
Low-medium
Low-medium
Query latency
Low on well-sized hot indices
Low if bounded by labels; worse on broad scans
Low-medium depending on cache and object storage
Very low on well-modeled aggregations
Scaling
Mature, but operationally involved
Horizontal and economical
Decouples compute and storage
Excellent for analytics and ingestion
Kubernetes
Good
Excellent
Good
Good, better with ClickStack/OTel
Grafana
Yes
Native
Yes
Yes
Elasticsearch compatibility
High in OpenSearch
No
Partial
No
Best use case
Complex search and compatibility
Low-cost Kubernetes logs
Efficient search on S3
Unified analytical observability
Main risk
Cost and complexity
Label cardinality
Ecosystem and governance after acquisition
Modeling and SQL/operations learning curve
Which Should You Choose in 2026?
The answer depends less on technology fashion and more on the team’s profile.
If You Are a Platform Team with Elasticsearch Heritage
OpenSearch is the conservative option if you already have automation, alerts and operational knowledge around Elasticsearch/OpenSearch. Do not migrate “as is” expecting costs to drop: review mappings, retention, ILM/ISM, shards, indexed fields and the separation between hot data and archive.
If You Are a Kubernetes Team Centered on Grafana
Loki is usually the first option. It is efficient, integrates very well with Grafana and reduces the cost of operational logs. The key is to govern labels from day one: allowed taxonomy, cardinality limits and context-driven queries before text searches.
If You Want to Reduce Cost Versus OpenSearch Without Losing Search
Quickwit deserves a serious PoC when Loki falls short on search and OpenSearch is expensive. The decision must include continuity: repository activity, releases, clients, community support, Grafana integration and Datadog’s position.
If You Want a Unified Observability Platform
ClickHouse is probably the strongest bet for consolidating logs, traces, derived metrics and events in a shared analytical database. ClickStack and HyperDX reduce the need to build the entire experience from scratch, but adoption depends on maturity in SQL, modeling and analytical database operations.
If You Have Security, Audit or SIEM Requirements
OpenSearch remains more natural for textual investigation and security tooling. ClickHouse can work for security analytics at scale, but it requires modeling. Loki is rarely the first choice for SIEM. Quickwit can be interesting, with a smaller ecosystem.
If Budget Is the Bottleneck
Do not choose only by storage cost per TB. Calculate total cost: ingestion, queries, retention, operations, backups, upgrades, training and incident time. Loki and ClickHouse usually win on raw cost; Quickwit can be highly competitive on object storage; OpenSearch can be reasonable if it is limited to data that is truly searchable and hot.
How to Frame a Decision PoC
A useful test is not installing four Helm charts and looking at a demo. Define real questions from past incidents and run them against representative data:
The last 7 days of production logs, with real cardinality.
A hot 2-hour window with high volume.
A rare text search over a broad period.
An aggregation by service, endpoint, version and region.
A correlation flow from alert to logs and traces.
Measure sustained ingestion, storage cost, p50/p95 latency, operational complexity and team experience. Include an uncomfortable scenario: a failed node, slow object storage, unexpected cardinality or a badly written query.
Conclusion
The era when Elasticsearch was the only viable option for logs is over. In 2026, there are four clear paths.
OpenSearch is the mature choice for full-text search and compatibility. Loki is efficient for operational logs in Kubernetes. Quickwit offers modern search on object storage, with mandatory governance evaluation after the Datadog acquisition. ClickHouse is consolidating as an analytical engine for unified observability, especially with ClickStack and HyperDX.
The best platform is not the one that wins every benchmark. It is the one that answers your production questions with the lowest sustainable cost and the least operational friction for your team.
CTA: Decide with Data, Not Preferences
Before migrating or standardizing, run a two-week PoC with real data, real queries and an honest estimate of total cost. It should reveal which data belongs in search, which belongs in analytics and which belongs in cheap storage.
Most “Kamal vs Kubernetes” articles are written by people who just discovered Kamal and are excited about it. They frame it as a simpler alternative that embarrasses Kubernetes. That framing is wrong and it will lead you to the wrong decision.
Kamal is not better than Kubernetes. Kubernetes is not overkill for everyone. They solve different problems at different scales, and picking the wrong one for your context is expensive either way.
This article gives you the honest version: what each tool actually is, where each one breaks, and the specific signals that should push you in one direction or the other.
What Kamal actually is
Kamal (formerly MRSK) is a deployment tool built by 37signals. It deploys Docker containers to servers via SSH, uses kamal-proxy as its default reverse proxy, and handles rolling deploys with zero downtime. That is the entire feature set.
kamal deploy
It connects to your servers over SSH, pulls the new image, starts the new container, waits for it to be healthy, then moves traffic through kamal-proxy and stops the old one. Traefik is still possible, but in Kamal 2 it sits in front of kamal-proxy as an optional accessory rather than being the standard path.
There is no control plane. No etcd. No API server. No scheduler. No concept of desired state reconciliation. Docker or systemd can restart a crashed container if you configure the right restart policy, but Kamal itself does not reschedule workloads to another node or provide cluster-level self-healing.
Kamal is a deployment tool. Kubernetes is an orchestration platform. These are not the same category.
That is the key distinction from tools like Nomad. I covered that angle in Nomad vs Kubernetes, as part of the same “lighter alternatives to Kubernetes” discussion. Nomad is a scheduler/orchestrator. Kamal is a deployment tool that leaves scheduling and node-level recovery mostly outside its scope.
What Kubernetes actually is
Kubernetes is a distributed system that continuously reconciles actual state against desired state. You declare what you want — N replicas of container X, with these resource limits, this health check, this rollout strategy — and the control plane makes it happen and keeps it that way.
When a node fails, your pods are rescheduled. When a container crashes, it is restarted. When you push a new image, the rollout respects your maxUnavailable and maxSurge settings. When traffic spikes, HPA can add replicas.
This machinery is powerful. It is also genuinely complex to operate.
A minimal production-grade Kubernetes cluster involves: a multi-node control plane for HA, a CNI plugin, an ingress controller (or Gateway API), cert-manager for TLS, a CSI driver for storage, a metrics server for HPA, RBAC policies, network policies, pod disruption budgets, and some form of secret management. Getting all of this right takes time and ongoing maintenance.
The question is whether the problems Kubernetes solves are problems you actually have.
The minimum config tells the story
Here is the same tiny web service expressed in both worlds. The exact fields vary by app, registry, ingress setup, and Kubernetes distribution, but the shape is representative.
env: secret: – DATABASE_URL</code></pre> </td> <td> <pre><code class=”language-yaml”>apiVersion: apps/v1 kind: Deployment metadata: name: web spec: replicas: 2 selector: matchLabels: app: web template: metadata: labels: app: web spec: containers: – name: web image: ghcr.io/acme/web ports: – containerPort: 3000 envFrom: – secretRef: name: web-secrets — apiVersion: v1 kind: Service metadata: name: web spec: selector: app: web ports: – port: 80 targetPort: 3000</code></pre> </td> </tr> </tbody> </table>
The Kubernetes example is still incomplete for internet traffic and TLS: you would normally add an Ingress or HTTPRoute, cert-manager, DNS, and secret provisioning. Kamal’s example is closer to “deploy this app to these servers.” Kubernetes is closer to “declare this workload inside a platform that already exists.”
The 37signals case
37signals (the company behind Basecamp and Hey) moved off cloud Kubernetes to bare-metal servers running Kamal in 2022-2023. Their own Ops team wrote about the “de-cloud and de-k8s” move. The story got a lot of attention.
What the hot takes missed: 37signals is not a startup. They have a dedicated ops team. They own and operate hardware. Their workload is well-understood and stable: a small number of mature applications serving a large, established user base.
Their decision was rational for their context. Kubernetes was costing them in cloud spend and operational complexity for problems their workload does not have. Moving to Kamal on owned hardware made economic and operational sense for them.
It does not follow that Kamal is the right choice for your early-stage startup, your company that runs on AWS because your team does not want to manage hardware, or your platform that needs to scale from 0 to 10x on short notice.
Where Kamal breaks
No cluster-level self-healing. If your container crashes, Docker or systemd can restart it with the right policy, but Kamal has no involvement. If your server fails, Kamal will not reschedule that workload elsewhere; your service is down until you intervene or until you have set up something else to handle it.
No built-in auto-scaling. Kamal has no equivalent of HPA or KEDA. If your traffic spikes, you add servers manually or script something yourself.
No multi-AZ HA by default. You can deploy to multiple servers across regions, but the coordination is yours to manage. There is no concept of spreading replicas across availability zones automatically.
Rolling deploys are basic. Kamal’s zero-downtime deploy is good for single-server or simple multi-server setups. It is not a substitute for Kubernetes rolling update with fine-grained health check integration across a fleet.
Secret management is manual. Kamal reads secrets from .kamal/secrets and has kamal secrets helpers for fetching and extracting values from password managers and cloud secret stores, but there is no Kubernetes-style projected secret lifecycle. Rotation and policy remain yours to manage.
No workload isolation. Everything on the same server shares the same kernel, same resources. Resource limits are Docker-level, not enforced by an orchestration layer with quotas and namespaces.
Where Kubernetes breaks
The complexity tax is real. Running Kubernetes yourself — not a managed service, but actual cluster operation — requires dedicated expertise. A team without a platform engineer will spend significant time debugging networking issues, understanding why pods are Pending, or figuring out why cert-manager is not issuing certificates.
Managed K8s is not free. EKS charges for the cluster control plane, with standard support priced around $0.10/hour per cluster; GKE and AKS have their own pricing rules and credits. In all cases, you still pay for nodes, load balancers, and persistent volumes. For a small application, this cost can exceed what bare-metal hosting would cost.
The YAML surface area scales with complexity. A simple web app on Kubernetes requires a Deployment, a Service, an Ingress (or HTTPRoute), a Certificate, potentially a HorizontalPodAutoscaler, a PodDisruptionBudget, and NetworkPolicies. That is a lot of infrastructure for a CRUD app.
Debugging is harder. Distributed systems are harder to debug than single-server setups. kubectl exec and kubectl logs are good tools but diagnosing why traffic is not reaching your pod — DNS, CNI, network policy, service selector, readiness probe — requires Kubernetes-specific knowledge that takes time to build.
The honest decision framework
Ask yourself these questions in order:
1. Do you have more than one team working on independent services with different scaling needs?
If yes: Kubernetes starts making sense. The isolation and per-service resource management are worth the overhead.
If no: Kamal is probably sufficient.
2. Do you need to survive individual server failures automatically, without manual intervention?
If yes: You need either Kubernetes with proper HA setup, or a managed container platform whose recovery and placement model fits your app. Cloud Run, Fly.io, Render, and Railway all help here, but their HA boundaries differ. Kamal alone does not give you this.
If no: A Docker or systemd restart policy covers the common crash case.
3. Does your traffic profile require automatic scaling?
If yes: Kubernetes with HPA or KEDA, or a managed platform with the scaling behavior you actually need. Cloud Run scales request-driven services to and from zero; Fly.io supports autostart/autostop and machine scaling; Render and Railway support scaling, but with provider-specific limits and billing. Kamal has no answer here without custom tooling.
If no: Fixed capacity is fine. Kamal works.
4. Do you have a platform team, or is “ops” a shared responsibility among product engineers?
If platform team exists: Kubernetes operational overhead is distributed and manageable.
If ops is everyone’s side job: Kamal’s simplicity is a genuine advantage. Less to go wrong, less to learn, faster to debug.
5. Are you on cloud infrastructure you do not own, and cloud costs matter?
If yes: Run the numbers on managed K8s vs a few VPS instances. For small workloads, VPS + Kamal is significantly cheaper.
If no: You have hardware already. Kamal is almost certainly the right choice.
Side-by-side
Kamal
Managed container platforms
Kubernetes
Examples
Kamal on VPS/bare metal
Cloud Run, Fly.io, Render, Railway
EKS, GKE, AKS, self-managed
Deployment model
SSH + Docker
Platform API / Git deploy / container deploy
API-driven, reconciliation loop
Self-healing
Docker/systemd restart policy only
Platform-managed restart and placement
Pod rescheduling, node failure recovery
Auto-scaling
Manual
Provider-specific; not all scale the same way
HPA, VPA, KEDA
HA
Manual (multiple servers)
Provider-managed, with platform limits
Available with a well-configured multi-node control plane (not free)
TLS
kamal-proxy with automatic HTTPS; Traefik optional
Usually built in
cert-manager + ingress controller
Secret management
.kamal/secrets + helper commands
Platform secrets
Secrets API, external-secrets, Vault
Observability
Standard Docker logging
Provider logs/metrics, export varies
Rich ecosystem (Prometheus, OTel, etc.)
Rollout control
Basic rolling deploy
Simple rollouts, provider-specific
Fine-grained (maxUnavailable, canary, etc.)
Learning curve
Low
Low to medium
High
Ops overhead
Low
Low
Medium to high
Right scale
1-20 services, stable load
Small to medium teams that want managed recovery/scaling
10+ services, variable load, or regulated
What the threshold actually looks like
Kamal is the right default if you are:
A team of 2–10 engineers shipping a web application
Running stable, predictable workloads
On a budget where managed K8s costs matter
Without a dedicated platform engineer
Kubernetes is the right choice when:
You have 10+ services with independent deployment cycles
You have a platform team that can absorb the operational complexity
You are in a regulated environment that benefits from K8s’s audit trails and RBAC
There is a middle tier worth mentioning: managed container platforms like Fly.io, Railway, Render, and Google Cloud Run. These give you some of Kubernetes’s operational benefits with closer to Kamal’s operational simplicity, but the details matter: HA, auto-scaling, scale-to-zero, regions, and billing differ by provider. For teams that need more than Kamal but do not want to operate K8s, this tier is often the right answer and gets underrepresented in the debate.
The real lesson from 37signals
The story is not “Kamal beat Kubernetes.” The story is: match your infrastructure to your actual operational profile, not to what is fashionable or what scales to Google’s size.
37signals evaluated what problems they actually had and picked the tool that solved those problems at the lowest operational cost. That is the right framework.
For most teams reading this, Kubernetes is probably the right answer eventually — when your system’s complexity justifies it. The mistake is adopting it before you are there, burning engineering time on infrastructure problems instead of product problems.
Kamal is a good tool for a specific stage and scale. Use it until you outgrow it. When you outgrow it, you will know — because the things Kubernetes solves will be actual problems you have, not hypothetical ones.
What to do next
Before you choose, write down your actual requirements in one page: number of services, expected traffic variance, failure tolerance, who owns ops, whether you own hardware, and what compliance needs are real today. Then run a small proof of concept with the simplest tool that satisfies those requirements. If Kamal plus Docker restart policies covers the failure modes you actually accept, start there. If you need automatic placement, autoscaling, and multi-zone recovery now, compare managed container platforms first, then Kubernetes.
FAQ
Can Kamal and Kubernetes coexist?
Yes. Some teams use Kamal for simple auxiliary services (internal tools, cron jobs, simple APIs) while running their core platform on Kubernetes. The tools are not mutually exclusive.
Is Kamal production-ready?
Yes. 37signals runs Basecamp and Hey on it. It handles zero-downtime deploys, TLS, and multi-server deployments reliably. The limitation is not reliability — it is feature scope.
What about Docker Swarm? Is it still relevant?
Docker Swarm fills a similar niche — simpler than Kubernetes, multi-host orchestration. Swarm mode is still documented and supported as part of Docker Engine for teams that want it as a production runtime, but its ecosystem is much smaller than Kubernetes and less active in the market. I would not pick it as the default for a new project unless the team already knows and wants Swarm.
Does Kamal work with any cloud provider?
Yes. Kamal only requires SSH access to a server and a container registry. It works with any VPS provider (Hetzner, DigitalOcean, OVH, AWS EC2, etc.) and any registry (Docker Hub, GitHub Container Registry, ECR, Harbor).
Is Kubernetes worth learning even if you use Kamal today?
Yes. Kubernetes is the dominant orchestration platform. Understanding it — even without operating it — makes you a better platform engineer and opens more career options. Learning Kamal does not preclude learning Kubernetes.
You have ten microservices running in Kubernetes. Each one validates JWTs, checks scopes, maintains sessions, and implements its own RBAC rules. One team uses jsonwebtoken v8, another uses a custom Go library, a third rolled their own HMAC check because “it was simple.” They all accept alg: none. Three accept RS256 and HS256 simultaneously.
This is not a security posture. This is a distributed security liability — and Kubernetes makes the problem worse, because the cluster creates an illusion of isolation that encourages teams to treat each Pod as a security boundary it was never designed to be.
The pattern of embedding authentication and authorization logic inside individual containers is one of the most pervasive anti-patterns in Kubernetes-based microservices. It feels like ownership and simplicity. It is, in practice, inconsistency at scale — and the blast radius of a single misconfiguration is your entire service portfolio.
Kubernetes provides the primitives to fix this at the infrastructure layer: Ingress controllers, the Gateway API, service mesh sidecars, admission webhooks, and workload identity via SPIFFE. None of these require a line of auth code inside your application containers.
This article explains why the anti-pattern exists, what’s wrong with it technically, and what the correct Kubernetes-native alternatives are — with concrete implementation guidance and references to the standards and incidents that validate the argument.
The Anti-Pattern: What It Looks Like
In-Application JWT Validation
Every service imports an auth library and validates tokens independently:
# Pattern seen in thousands of microservices
from jose import jwt
def authenticate(token: str):
payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
return payload["sub"]
Variations include:
Algorithm confusion: accepting both HS256 and RS256, or letting the token header drive verification behavior instead of pinning acceptable algorithms server-side. This is a distinct JWT implementation failure class, documented extensively by PortSwigger on JWT attacks and RFC 8725
alg: none bypass: libraries that accept unsigned tokens when alg is set to none. This is a documented attack vector in Auth0’s JWT security analysis
Missing exp / iss / aud validation: trusting a valid signature without checking whether the token is expired, for the right audience, or from the right issuer
Key confusion attacks: accepting a public RS256 key as an HS256 symmetric secret
Session State in Every Service
When services maintain session state directly, they duplicate logic that has no business being duplicated — cookie validation, refresh token flows, PKCE verification — and each implementation diverges over time.
RBAC Reimplemented Per Service
Authorization rules (“can this user access this resource?”) end up embedded in service logic, mixed with business logic, tested inconsistently, and impossible to audit across the portfolio.
Why This Is a Structural Problem
1. The Vulnerability Surface Scales With Your Service Count
Each new microservice is a new JWT validation surface. A single incorrect library configuration — an unvalidated alg, a missing aud check — is an authentication bypass affecting that entire service. With ten services, you have ten potential misconfigurations. With a hundred services, the probability that at least one is misconfigured approaches certainty.
When a JWT vulnerability is disclosed — and they are disclosed regularly — you update one library in one service. Then another. Then you discover service C is pinned to an old version because it has a transitive dependency conflict. Meanwhile the vulnerability is exploitable in production.
The CNCF Cloud Native Security Whitepaper frames this directly: security controls implemented redundantly across services create maintenance overhead that teams cannot sustain, leading to version drift and policy divergence.
3. Centralized Policy Is Impossible to Enforce
When policy is in code — even well-factored library code — it cannot be changed atomically across services. A policy update requires a coordinated deployment across every affected service. In practice, services deploy on different schedules, managed by different teams, with different testing cycles. The result is that at any given moment, some fraction of your services are running different authorization rules.
This is the core argument in Google’s BeyondCorp model and the Zero Trust Architecture guidance from NIST SP 800-207: authentication and authorization decisions should be made by a centralized, auditable policy enforcement point — not distributed across workloads.
4. Secrets Distribution Is a Problem You Don’t Need
If every service validates JWTs, every service needs the signing key (for symmetric algorithms) or the public key (for asymmetric). Distributing and rotating signing keys across a fleet of microservices is an operational burden with meaningful blast radius: a leaked symmetric key compromises every service holding it.
The CNCF SPIFFE/SPIRE project was built specifically to solve this class of problem: workload identity should be cryptographically attested, not rely on secrets distributed to application code.
The Real-World Incidents
The alg: none Class
In 2015, critical vulnerabilities in JWT libraries from Auth0 affecting Python, PHP, Node.js, Ruby, Java, and .NET allowed attackers to forge tokens by setting alg: none. The signature was not verified. The vulnerability was present in applications that had copied JWT validation code from tutorials or used unpatched libraries — exactly the pattern that in-service auth produces at scale.
Java’s Psychic Signatures (CVE-2022-21449)
CVE-2022-21449 affected ECDSA signature verification in Oracle Java SE and GraalVM, including java.security.Signature paths used by higher-level libraries. The bug allowed certain malformed ECDSA signatures to verify when they should not. JWT validation was in scope only when the deployment used an affected Java runtime and ECDSA-signed tokens, for example ES256. A gateway would help only if verification happened on a patched or unaffected runtime at the gateway instead of inside every Java service.
CVE-2023-2728 was not an ImagePolicyWebhook outage behavior. It was a Kubernetes API server issue where users could use ephemeral containers to bypass the mountable secrets policy enforced by the ServiceAccount admission plugin. Clusters were affected only when the ServiceAccount admission plugin, the kubernetes.io/enforce-mountable-secrets annotation, and ephemeral containers were used together. The adjacent ImagePolicyWebhook issue was CVE-2023-2727, also involving ephemeral containers, but it is a separate CVE.
The Uber API Gateway Evolution
Uber’s engineering blog describes its API gateway as a centralized layer for routing, protocol conversion, rate limiting, load shedding, header propagation, security auditing, and user access blocking. That supports the architectural point here: high-volume platforms move cross-cutting controls into shared infrastructure. The public source does not prove that Uber migrated specifically from per-service authentication to gateway authentication, so that stronger claim should not be made.
Netflix Zuul
Netflix’s Zuul is an L7 gateway for dynamic routing, monitoring, resiliency, security, and related edge concerns. Netflix’s own Zuul posts list authentication among common edge-service uses, but they do not frame Zuul primarily as a case study in eliminating per-service auth. Treat it as evidence that authentication is a natural edge concern at scale, not as proof of a specific migration story.
The Alternatives
Use these as complementary controls, not as a single replacement for all authentication and authorization logic:
Alternative
When to use it
What it solves
Principal trade-off
API Gateway / Edge Auth
External API clients, public ingress, partner integrations, mixed auth mechanisms at the boundary
Central JWT/API-key/OAuth2 validation, rate limiting, request shaping, and identity header propagation before traffic reaches services
Does not secure east-west service calls by itself; trusted headers require strict network boundaries
Service Mesh mTLS
Service-to-service traffic inside the cluster, especially across teams or sensitive domains
Workload identity, automatic mTLS, peer authentication, and proxy-level authorization policy
Adds data-plane/control-plane complexity and operational coupling to sidecars or ambient mesh components
OAuth2 Proxy
Browser-facing internal apps that need OIDC login, redirects, cookies, and session handling
Delegates login/session management to a reverse proxy and forwards authenticated identity headers
Best for HTTP/browser flows; not a general machine-to-machine authorization system
Separates policy decisions from application releases and can run as sidecar, service, ext_authz backend, or admission control via Gatekeeper
Policy/data distribution and failure behavior must be designed deliberately
SPIFFE/SPIRE
Multi-cluster, multi-cloud, or meshless environments that need portable workload identity
Issues short-lived workload identities without application-managed shared secrets
Provides identity, not business authorization; needs registration and attestation lifecycle management
Option 1: API Gateway (Edge Auth)
An API Gateway sits at the perimeter and handles authentication before a request reaches any downstream service. Services receive pre-validated identity in a trusted header.
What it does: – Validates JWTs, API keys, OAuth2 tokens – Enforces rate limiting per identity – Strips and re-adds Authorization headers as needed – Routes to upstream services with verified identity headers
When to use it: – North-south traffic (external clients → cluster) – Mixed authentication mechanisms (JWT + API key + mTLS) at the Ingress layer – Teams that want to centralize auth policy without rolling out a full service mesh
Gravitee’s Kubernetes operator reconciles ApiV4 resources against the gateway, making API policy a first-class GitOps object — versioned, reviewed, and deployed the same way as any other Kubernetes manifest.
Other Kubernetes-native options: Emissary-Ingress (formerly Ambassador) with its AuthService CRD; Traefik with ForwardAuth middleware on IngressRoute resources.
Limitations: API Gateways handle north-south traffic. They don’t address east-west (service-to-service) authentication inside the cluster.
Option 2: Service Mesh (East-West mTLS + Auth)
A service mesh provides mutual TLS between every service pair and enforces authorization policy at the sidecar proxy, without any application code changes.
What it does: – Automatic mTLS between all service-to-service calls – Workload identity via X.509 certificates (SPIFFE SVIDs) – Fine-grained AuthorizationPolicy at the Envoy sidecar – JWT validation at the proxy, not the application
The RequestAuthentication policy tells Envoy how to validate JWTs. The AuthorizationPolicy specifies what authenticated principals are allowed to do. Neither policy lives in application code.
The payment service receives the validated request — or a 401/403 from the proxy, before the request touches application code.
Linkerd:
Linkerd provides automatic mTLS with SPIFFE-compliant workload identity. Its policy model is simpler than Istio but sufficient for most service-to-service auth requirements:
OAuth2 Proxy is a reverse proxy that authenticates requests against an OAuth2/OIDC provider and passes validated identity downstream. With 14,000+ GitHub stars and active maintenance, it is the most widely deployed solution for this pattern in Kubernetes.
What it does: – Sits in front of one or more upstream services – Redirects unauthenticated requests to an OIDC provider (Keycloak, Dex, Google, GitHub, etc.) – Validates tokens, manages sessions, handles refresh – Passes X-Auth-Request-User, X-Auth-Request-Email, X-Auth-Request-Groups headers downstream
The upstream service receives X-Auth-Request-User and X-Auth-Request-Groups as trusted headers — it never sees a token, never validates a signature, never imports a JWT library.
When to use OAuth2 Proxy vs a service mesh: OAuth2 Proxy handles north-south browser-facing traffic with session management (login flows, redirects, cookies). A service mesh handles east-west machine-to-machine auth. They are complementary, not alternatives.
Option 4: Open Policy Agent (OPA / OPAL)
OPA decouples policy from code entirely. Authorization logic is written in Rego and evaluated by OPA as a sidecar or as a centralized service. Applications query OPA for allow/deny decisions.
# Rego policy — payment service authorization
package payments.authz
import future.keywords.if
import future.keywords.in
default allow := false
allow if {
input.method == "POST"
input.path == "/v1/payments"
"payments:write" in input.token.scope
input.token.iss == "https://accounts.example.com"
}
allow if {
input.method == "GET"
startswith(input.path, "/v1/payments/")
"payments:read" in input.token.scope
}
Application code becomes:
// The ONLY auth code in the application
func authMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
input := map[string]interface{}{
"method": r.Method,
"path": r.URL.Path,
"token": extractToken(r),
}
result, err := opaClient.Decision(r.Context(), "payments/authz", input)
if err != nil || !result.Allow {
http.Error(w, "Forbidden", http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
})
}
OPAL (Open Policy Administration Layer) adds real-time policy and data updates to OPA deployments — policy changes propagate to all OPA instances within seconds without redeployment.
Kubernetes deployment patterns for OPA:
As a sidecar — OPA runs in the same Pod as the application, evaluating policy over a local socket. Zero network hop, no external dependency:
As a centralized service with Envoy ext_authz — OPA exposes a gRPC endpoint that Istio’s Envoy sidecar calls for every request. Policy is enforced at the proxy, before the application receives the request. This is the pattern used alongside Istio’s EnvoyFilter shown in the service mesh section above.
As OPA Gatekeeper — OPA Gatekeeper runs as a Kubernetes admission webhook and enforces policies at deploy time, not at runtime. It’s the right tool for preventing misconfigured workloads from being deployed — for example, rejecting any Pod spec that sets hostNetwork: true or defines auth-related environment variables directly. This is complementary to runtime auth enforcement.
SPIFFE (Secure Production Identity Framework for Everyone) and SPIRE solve the problem of how workloads prove their identity without distributing secrets.
SPIRE issues short-lived X.509 SVIDs (SPIFFE Verifiable Identity Documents) to workloads. Each SVID encodes a SPIFFE URI:
Services authenticate each other using mTLS with these certificates. No JWT library. No shared secret. No secret distribution problem. Certificate rotation happens automatically every few hours.
SPIFFE is the foundation of Istio’s workload identity model. Linkerd implements SPIFFE-compatible SVIDs. If you’re using a service mesh, you already have SPIFFE — the mesh uses it transparently.
Standalone SPIRE is appropriate for environments without a service mesh, or for multi-cluster/multi-cloud scenarios where a consistent workload identity layer is needed across boundaries.
SPIFFE/SPIRE graduated from the CNCF sandbox to incubating in 2019 and is deployed at Uber, Bloomberg, ByteDance, and Anthem.
Combining the Layers
These tools are not mutually exclusive — they address different traffic patterns and different problems:
Layer
Tool
Addresses
Edge (north-south)
API Gateway (Gravitee, Emissary, AWS APIGW)
External clients → cluster
Browser sessions
OAuth2 Proxy
Browser-facing apps with login flows
East-west mTLS
Service Mesh (Istio/Linkerd) + SPIFFE
Service-to-service identity
Policy
OPA/OPAL
Fine-grained, auditable authorization
Workload identity
SPIRE
Multi-cloud/multi-cluster identity
A production deployment at reasonable scale looks like:
External traffic hits an API Gateway or Ingress controller with OAuth2 Proxy
The gateway validates the token, strips it, and forwards identity headers to the upstream service
Inside the cluster, all service-to-service calls are mTLS via a service mesh, using SPIFFE workload identity
Authorization decisions (beyond identity) are delegated to OPA
Application code contains zero JWT validation, zero session management, zero auth library imports
Migration Path
If you already have auth embedded in services, migration doesn’t require a big bang rewrite.
Phase 1: Introduce the Gateway
Deploy an API Gateway or OAuth2 Proxy at the edge. Initially, services continue to validate tokens themselves as a backup — the gateway validates first. Use this phase to verify the gateway’s behavior and build confidence.
Phase 2: Trust the Gateway
Add a service-level feature flag: if a trusted X-Auth-Request-User header is present (set by the gateway), skip internal JWT validation. This decouples service auth from gateway rollout.
Phase 3: Remove In-Service Auth
Once all entry points are covered by the gateway and you have confidence in its reliability, remove the auth code from services. This is the step that actually reduces your attack surface.
Phase 4: Add East-West (Optional)
If east-west service-to-service calls exist and carry sensitive data, introduce a service mesh for mTLS. This is a separate effort from gateway auth and can proceed independently.
Decision Framework
External traffic entering the cluster (Ingress / Gateway API)?
├── Browser-facing app with login flow → OAuth2 Proxy (Nginx Ingress annotations)
└── API clients with tokens → API Gateway (Gravitee ApiV4 / Emissary AuthService)
Service-to-service calls inside the cluster (east-west)?
├── mTLS + identity sufficient → Service Mesh (Istio/Linkerd)
├── Identity across multiple clusters → SPIFFE/SPIRE standalone
└── Istio + fine-grained policy → RequestAuthentication + OPA via ext_authz
Complex, auditable authorization logic?
└── OPA as sidecar or ext_authz (runtime) + OPA Gatekeeper (admission)
Preventing misconfigured workloads from being deployed?
└── OPA Gatekeeper admission webhook
What to Keep in Application Code
Not everything should be removed. The correct model is:
Keep: Structured logging of authorization decisions for audit trails
The application receives an authenticated identity from infrastructure. What the application does with that identity — which records to show, which operations to allow based on ownership — is correctly application logic.
Audit Checklist: Moving Auth Out of Containers
Use this as a practical exit checklist for the anti-pattern:
Inventory every service that imports JWT, OAuth2, OIDC, session, or custom RBAC libraries.
Classify each entry point as north-south, browser session, east-west service call, deploy-time admission policy, or business authorization.
Put one enforcing control in front of each class: API Gateway or OAuth2 Proxy for ingress, service mesh mTLS for east-west, OPA for shared policy, and SPIFFE/SPIRE for portable workload identity.
Pin JWT issuers, audiences, algorithms, and JWKS sources in infrastructure policy; do not let application code infer them from token headers.
Strip client-supplied identity headers at the edge and re-add trusted identity headers only after verification.
Define failure behavior explicitly: fail closed for authentication and authorization, and document any temporary fail-open exception with an owner and expiry date.
Remove in-service token verification only after every ingress path is covered, logs prove the infrastructure control is enforcing, and rollback has been tested.
Keep resource-level business authorization in application code, but feed it an identity established by infrastructure.
Running container images inside Kubernetes is table stakes. Building them there — or mounting their contents as volumes — has been a moving target for years. What started as a privileged hack has evolved into a set of mature, secure, and increasingly native primitives.
This article covers the full arc: why the original approach was broken, what Kaniko solved, why its maintenance status now matters, where BuildKit and Buildah fit, and what Image Volumes (stable in Kubernetes v1.36) change for workflows that don’t need to build anything at all.
The original problem: Docker-in-Docker
The first generation of CI/CD on Kubernetes ran Docker inside Docker. You mounted the host Docker socket (/var/run/docker.sock) into a build container, and that container had full access to the host’s Docker daemon.
# The approach nobody should be using in 2026
volumes:
- name: docker-sock
hostPath:
path: /var/run/docker.sock
This worked. It was also a complete security disaster.
Mounting the Docker socket gives the container root-equivalent access to the host. Any workload that can reach that socket can escape the container, inspect other containers, and compromise the node. The only thing standing between your CI pipeline and a full cluster compromise was the good intentions of whoever wrote the build script.
It got worse when the Kubernetes project deprecated Dockershim in v1.20 and removed it in v1.24. Clusters that moved to containerd or CRI-O no longer had a Docker daemon on the host at all. The socket either didn’t exist or belonged to a completely different runtime. Docker-in-Docker in its classic form became functionally impossible on modern clusters.
Kaniko: daemonless builds inside containers
Google released Kaniko in 2018 to solve exactly this problem. Kaniko builds container images entirely in userspace — no daemon, no privileged access, no host socket required.
The key insight: Docker builds work by executing each RUN instruction in a temporary container, snapshotting the filesystem, and saving the result as a layer. Kaniko replicates this logic without a daemon. It runs as a regular container, reads the Dockerfile, executes each step against the local filesystem, and pushes the resulting image directly to a registry.
That design aged well. The project governance did not. The GoogleContainerTools/kaniko repository was archived by its owner on June 3, 2025 and is now read-only. In practical terms, the original Google-hosted Kaniko project should be treated as unmaintained: no active upstream issue triage, no normal pull request flow, and no clear path for security fixes through that repository.
No host socket. No privileged flag. The container needs write access to its own filesystem (so readOnlyRootFilesystem: true won’t work), but that’s a far narrower requirement than socket mounting.
When Kaniko is still a reasonable answer
Kaniko can still be a reasonable tactical choice when you need to build a container image inside Kubernetes and already have a working, isolated pipeline around it. Existing Kaniko jobs did not stop working when the repository was archived.
For new platform work in 2026, though, do not pick Kaniko by default. The maintenance signal changes the risk model. Use it only if the simplicity is worth owning the upgrade and vulnerability-management story yourself, or if you deliberately standardize on a maintained fork or vendor-supported distribution.
Kaniko integrates well with: – Tekton — the standard pattern is a Tekton Task running the Kaniko executor – Argo Workflows — same pattern, different orchestrator – GitLab CI on Kubernetes — many older examples and pipelines use Kaniko with the Kubernetes executor – Any pod-based CI system — it’s just a container, so it runs anywhere pods run
The alternatives: BuildKit and Buildah
Kaniko is no longer the only serious daemonless option. Two other tools are worth evaluating first for new work:
BuildKit (rootless mode)
BuildKit is the build backend behind docker buildx and the default build system in modern Docker. In rootless mode it runs without privileges and can build images inside a Kubernetes pod.
BuildKit has better caching than Kaniko — particularly layer caching via cache mounts — and supports more advanced Dockerfile features like heredocs and multi-platform builds. The tradeoff is more complex setup: you need to run buildkitd as a sidecar or as a DaemonSet.
For teams already using docker buildx locally, BuildKit is the most natural migration path. Docker’s Kubernetes driver can run BuildKit builders directly in a cluster, including rootless mode without privileged pods on supported Kubernetes versions. The operational cost is real — builder lifecycle, cache persistence, node placement, and rootless kernel requirements — but the project is active and the feature set is where most modern Dockerfile workflows are moving.
Buildah
Buildah is the containers project’s daemonless build tool, designed to integrate with Podman and OCI-native workflows. It can build from Dockerfiles or Containerfiles, and it is the natural choice on OpenShift or in environments where the platform team already standardizes on Red Hat, Podman, and the containers/* stack.
The caveat is that rootless Buildah inside a restricted Kubernetes pod still depends on user namespace behavior and helper binaries such as newuidmap and newgidmap. That is manageable on platforms built for it, especially OpenShift, but it is not automatically a drop-in replacement for every generic Kubernetes CI runner.
Image Volumes: a different problem entirely
Here is where the narrative splits. Everything above is about building images. Image Volumes solve a completely different problem: consuming the contents of an OCI image as a volume, without building anything.
The feature was introduced as alpha in Kubernetes v1.31, moved to beta in v1.33 with subPath and subPathExpr support, became beta enabled by default in v1.35, and graduated to stable (GA) in Kubernetes v1.36.
What it does
Image Volumes let you reference an OCI image in a pod’s volumes section and mount its filesystem contents directly into a container:
This feature is motivated by a pattern that has been growing quietly: using OCI images not as runnable containers but as versioned, signed, distributable artifact bundles.
The idea: instead of storing configuration, schemas, WASM modules, ML models, or static binaries in a ConfigMap or a separate volume, you package them as an OCI image. You get:
Version control — image tags and digests, same tooling you already use
Distribution — your existing registry, your existing pull secrets, your existing access controls
Signing and attestation — cosign, Sigstore, the full supply chain tooling works on these artifacts
Immutability — a digest-pinned image reference is cryptographically immutable
Image Volumes are the Kubernetes primitive that makes this pattern first-class. Without Image Volumes, the workaround was an init container that pulled the image and copied the contents to an emptyDir. It worked, but it was boilerplate.
What it does not do
Image Volumes are not a replacement for Kaniko or any build tool. They consume images; they don’t produce them. If your workflow involves building a new image from source, you still need Kaniko, BuildKit, or equivalent.
They also require container runtime support. CRI-O supported the initial alpha implementation from its v1.31 line and tracked beta support for v1.33; containerd support landed later through the containerd 2.x line. On anything before Kubernetes v1.36, verify both the Kubernetes feature gate and the runtime version before treating this as production plumbing.
Decision framework by Kubernetes version
K8s version
Build images (CI)
Mount image contents as volume
< 1.24
Avoid Docker socket builds; use BuildKit, Buildah, or legacy Kaniko with explicit risk acceptance
Init container + emptyDir workaround
1.24 – 1.30
BuildKit or Buildah preferred; legacy Kaniko only if already standardized
Init container + emptyDir workaround
1.31 – 1.32
BuildKit or Buildah preferred; legacy Kaniko only with maintenance plan
BuildKit or Buildah preferred; legacy Kaniko only with maintenance plan
Image Volumes beta with subPath/subPathExpr, but disabled by default; enable ImageVolume and verify runtime support
1.35
BuildKit or Buildah preferred; legacy Kaniko only with maintenance plan
Image Volumes beta, enabled by default; still verify runtime support before production rollout
1.36+
BuildKit or Buildah preferred; legacy Kaniko only for existing pipelines or maintained forks
Image Volumes stable (GA), enabled by default
When to use what
Use Kaniko if: – You already have working Kaniko pipelines and the operational cost of migration is higher than the current risk – You have a maintained fork, vendor support, or an internal patching process – You want the simplest daemonless build setup and accept that upstream Google Kaniko is archived
Use BuildKit (rootless) if: – You need advanced cache mounts or multi-platform builds – Your team already uses docker buildx locally – You’re willing to run and operate BuildKit builders in the cluster
Use Buildah if: – You are on OpenShift or a Podman/Red Hat-oriented platform – You want an OCI-native build tool that does not require a daemon – Your cluster policy supports the user namespace requirements of rootless builds
Use Image Volumes if: – You want to inject versioned, signed artifacts into pods without building anything – You’re replacing init container + emptyDir patterns – You’re adopting OCI images as a general artifact format (configs, schemas, binaries)
Conclusion
The container image story inside Kubernetes has matured significantly. Docker-in-Docker is dead — correctly so. Kaniko solved an important build problem in 2018, but the original Google project is archived in 2026, so it should no longer be the default recommendation for new platforms. BuildKit and Buildah are the healthier starting points for active build pipelines, with Kaniko reserved for existing estates or explicitly supported forks.
Image Volumes are genuinely new ground. They’re not competing with Kaniko — they’re addressing a different layer of the same ecosystem: the distribution and consumption of OCI artifacts beyond just “images you run.” With GA in v1.36 and the supply chain tooling around OCI reaching maturity, the pattern of “package it as an image, distribute it like an image, mount it where you need it” is becoming the right answer for a class of problems that previously lived in ConfigMaps, PVCs, or init container hacks.
The right combination depends on what you’re doing and what version you’re running. But for the first time, Kubernetes has native answers for both sides of the equation.
Choose today
If you operate Kubernetes v1.36 or newer, use Image Volumes for read-only artifact injection and choose BuildKit or Buildah for builds. If you are on v1.35, Image Volumes are beta and enabled by default, but you should still verify runtime support and keep the init-container pattern as the rollback path. If you are on v1.33 or v1.34, beta does not mean default-on: enable the ImageVolume feature gate deliberately and validate the runtime. If you are on v1.31 or v1.32, treat Image Volumes as alpha and non-default. If you are below v1.31, Image Volumes are not part of the platform: use emptyDir plus an init container for artifact mounting, and modernize the build path separately.
For CI builds, start with BuildKit when you want Dockerfile compatibility, cache performance, multi-platform output, and a path aligned with docker buildx. Start with Buildah when your cluster is OpenShift or Podman-oriented. Keep Kaniko only where it already works and where someone owns the maintenance risk.
EKS Auto Mode is Amazon EKS with AWS running the node layer for you: it provisions EC2 instances with Karpenter-style logic, patches and replaces them on a fixed lifetime, and runs networking, DNS, block storage and load balancing as managed capabilities instead of add-ons you install. You pay for it with a management fee per EC2 instance-hour, on top of the normal EC2 price, and you give up node-level access and custom AMIs.
That is the whole trade in one sentence. The rest of this guide is about whether it is a good trade for your cluster: how the pricing actually works (with a worked example), how Auto Mode compares to managed node groups, self-managed Karpenter and Fargate, what changed in 2025-2026, what breaks when you migrate, and a one-week pilot to decide without risking production.
What EKS Auto Mode is
EKS Auto Mode, generally available since December 1, 2024, shifts more Kubernetes infrastructure responsibility from you to AWS. You still run an EKS cluster in your AWS account, and your workloads still use the Kubernetes API, but AWS takes over much of the compute, storage, networking, node lifecycle, and core add-on management that platform teams usually wire together themselves.
For compute, Auto Mode uses Karpenter under the hood. When pods are unschedulable, Auto Mode provisions nodes that fit the workload’s requirements: instance family, size, architecture, capacity type, and availability zone. When capacity is no longer useful, it consolidates and terminates nodes. The nodes run a locked-down variant of Bottlerocket with SELinux enforcing and a read-only root filesystem, and there is no SSH or SSM access.
The important framing is this: Auto Mode is not “EKS without nodes.” It is EKS where AWS manages the node lifecycle more aggressively. You own the workloads, their scheduling requirements, their disruption behavior, and the operational consequences of those choices. AWS owns more of the infrastructure plumbing.
What it replaces
Before Auto Mode, running EKS in production usually meant choosing and operating several layers yourself:
Managed node groups: you chose instance types, defined scaling ranges, managed AMI updates, handled node draining, and configured Cluster Autoscaler or another scaling mechanism.
Self-managed Karpenter: more flexible than managed node groups, but you owned the Karpenter controller, IAM, NodePools, EC2NodeClasses, the Spot interruption queue, disruption settings, upgrades, and failure modes.
Fargate: AWS-managed compute per pod, with no node management, but no DaemonSets, a narrower workload compatibility envelope, and a different cost model.
EKS Auto Mode replaces a large part of that platform assembly with a managed model: declare workload intent and high-level compute constraints; AWS provisions and manages the EC2 instances behind it. If you are still deciding between the two open-source autoscalers, see Cluster Autoscaler vs Karpenter first; Auto Mode is essentially “Karpenter, operated by AWS”.
EKS Auto Mode pricing: how the fee actually works
This is the part most people search for, and the part most summaries get slightly wrong. The official structure, from the Amazon EKS pricing page (checked September 2026), is:
Monthly cost of an EKS Auto Mode cluster =
EKS control plane ($0.10 per cluster-hour, standard support)
+ EC2 instances (normal price: On-Demand, Spot, RI or Savings Plans)
+ EKS Auto Mode management fee (per instance-hour, by instance type and region)
+ everything else (EBS, load balancers, NAT, data transfer, CloudWatch...)
Auto Mode fee = Σ (Auto Mode-managed instance runtime × regional fee for that instance type)
Four details matter more than the headline:
The fee is per instance, not per cluster. It is charged only on EC2 instances that Auto Mode launches and manages. A cluster that runs a mix of Auto Mode nodes and legacy managed node groups only pays the fee on the Auto Mode ones.
It is billed like EC2: per second, with a one-minute minimum.
It is independent of how you buy the EC2 capacity. On-Demand, Spot, Reserved Instances and Compute Savings Plans all work with Auto Mode, but the management fee does not get discounted with them. This is the single most important pricing detail, and we will see why in the example below.
It varies by instance type and region. AWS does not publish it as a contractual percentage. In the public examples for US West (Oregon) the fee works out to exactly 12% of the On-Demand rate, but always pull the real per-type rate for your region before modelling anything.
The published Oregon examples are:
Instance
EC2 On-Demand ($/h)
Auto Mode fee ($/h)
Fee as % of On-Demand
m5a.xlarge
0.172
0.02064
12%
m5a.2xlarge
0.344
0.04128
12%
c6a.2xlarge
0.306
0.03672
12%
c6a.4xlarge
0.612
0.07344
12%
GPU and accelerated instances have their own fee schedule, and AWS cut it on July 1, 2026: management fees for G-series instances dropped by 35%, and for P-series and Trainium by 60%, automatically for every Auto Mode cluster. If your earlier cost model for an ML cluster was built in 2025, redo it.
Worked example: 10 × m5a.2xlarge for a month
Take a steady cluster of ten m5a.2xlarge nodes in us-west-2, running 730 hours a month:
Line item
Managed node groups
EKS Auto Mode
Control plane
$73.00
$73.00
EC2 On-Demand (10 × $0.344 × 730)
$2,511.20
$2,511.20
Auto Mode fee (10 × $0.04128 × 730)
—
$301.34
Total
$2,584.20
$2,885.54
On raw bill, Auto Mode is about +11.7% for this cluster. Now apply discounts to the EC2 part only, because that is what happens in reality (the discount levels below are illustrative, not quotes):
EC2 purchase option
EC2 cost
Auto Mode fee
Fee relative to your EC2 bill
On-Demand
$2,511.20
$301.34
12%
Savings Plan at ~30% off
$1,757.84
$301.34
~17%
Spot at ~65% off
$878.92
$301.34
~34%
The better you already are at buying compute, the more expensive Auto Mode looks in relative terms, because the fee is anchored to the On-Demand-scale price and does not shrink with your discount. A team that runs mostly Spot pays the equivalent of roughly a third on top of its compute bill.
The other side of the ledger is efficiency. If Auto Mode’s consolidation lets the same workloads run on nine nodes instead of ten, the Auto Mode bill becomes $2,260.08 EC2 + $271.21 fee + $73 = $2,604.29, within 1% of the managed node group cluster. In other words, you need roughly 11-12% better bin packing to break even on the raw bill at On-Demand prices. Clusters with fixed-shape node groups sized “just in case” often waste more than that; clusters already on well-tuned Karpenter usually do not.
Then add the part the bill does not show. The fee on this cluster is about $3,600 a year. At 100 nodes it is about $36,000 a year. Compare that against the hours your team spends on AMI rotation, autoscaler upgrades, add-on version matrices and node incidents. For a two-person platform team the fee is usually cheap; for a large platform team that already automated all of it, it is usually not.
Fargate for comparison
Fargate is priced per pod, per second, on requested vCPU and memory. In US East (N. Virginia), Linux/x86 is about $0.04048 per vCPU-hour and $0.004446 per GB-hour (September 2026), so a 1 vCPU / 2 GB pod running all month costs about $36. There is no node to pay for, but also nothing to bin-pack: you pay for requests, not usage, which makes right-sized requests and limits even more important than on EC2. And Amazon EKS does not support Fargate Spot.
EKS Auto Mode vs managed node groups vs Karpenter vs Fargate
EKS Auto Mode
Managed node groups
Self-managed Karpenter
Fargate
Who runs node provisioning
AWS (managed Karpenter)
You (ASG + Cluster Autoscaler)
You (Karpenter controller)
AWS, per pod
Instance selection
Dynamic via NodePools
Fixed list per node group
Dynamic via NodePools
Not exposed
Node OS / AMI
Bottlerocket variant, chosen by AWS
AL2023, Bottlerocket, Windows or custom
AL2023, Bottlerocket, Windows or custom
Not exposed
AMI patching
Automatic, weekly AMIs, 21-day max node lifetime
You roll updates
You set expireAfter and drift
N/A
SSH / SSM to nodes
No
Yes, if enabled
Yes, if enabled
No
DaemonSets
Yes, but no host changes
Yes
Yes
No
Windows
No
Yes
Yes
No
Spot
Yes, interruptions handled natively
Yes
Yes, you configure the SQS interruption queue
No
Networking, DNS, EBS CSI, LB controller
Managed capabilities
Add-ons you install and upgrade
Add-ons you install and upgrade
Managed, with Fargate limits
Extra cost
Management fee per instance-hour
None
None (controller runs on your capacity)
Per-pod vCPU/GB pricing
Operational burden
Low for nodes, medium for workload compatibility
Medium
Medium-high
Low for nodes, medium for compatibility
Right for
Teams that don’t need node customization
Regulated/custom node environments
Mature platform teams that want full control
Isolated, simple workloads
EKS Auto Mode vs managed node groups
This is the comparison most teams actually face. Managed node groups give you a fixed capacity shape and full control of the node: your AMI, your kubelet flags, your host agents. Auto Mode gives you dynamic capacity and removes node operations, but takes the node away from you.
Pick managed node groups if you need custom or hardened AMIs, Windows, host-level security tooling, specific kernel settings, or if your capacity is steady and already heavily discounted (the fee hurts most there). Pick Auto Mode if your node groups are over-provisioned, your team spends real time on AMI rotations and add-on upgrades, and your workloads tolerate nodes being replaced at least every 21 days.
The two can coexist in the same cluster, which is also how migrations work: new workloads on Auto Mode, legacy or special ones on node groups.
EKS Auto Mode vs Karpenter
Auto Mode is Karpenter, but not your Karpenter. Both share the karpenter.sh/v1 NodePool API; the node class differs (NodeClass in eks.amazonaws.com/v1 for Auto Mode, EC2NodeClass in karpenter.k8s.aws/v1 for self-managed), and so do the instance labels (eks.amazonaws.com/instance-family vs karpenter.k8s.aws/instance-family).
Self-managed Karpenter has no fee and lets you choose the AMI, OS (including Windows), and node lifetime. In exchange you operate the controller, its IAM, the Spot interruption queue, GPU drivers and device plugins, and every upgrade. Auto Mode handles Spot interruptions without an SQS queue, ships NVIDIA and Neuron drivers, enables node repair by default, and upgrades the controller for you. If you already run Karpenter well, the fee buys you little. If you were about to adopt Karpenter, Auto Mode removes most of the adoption cost.
How it works in practice
You create or update an EKS cluster with Auto Mode enabled. The default setup uses two AWS-managed built-in node pools, system and general-purpose, which you can enable or disable but not modify. If you need more control, you create a NodeClass for Auto Mode infrastructure settings and a NodePool for workload-facing scheduling constraints.
# NodePool: workload-facing constraints for nodes that Auto Mode may provision.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: general-purpose-custom
spec:
template:
metadata:
labels:
billing-team: platform
spec:
nodeClassRef:
group: eks.amazonaws.com
kind: NodeClass
name: private-compute
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand", "spot"]
- key: kubernetes.io/arch
operator: In
values: ["amd64", "arm64"]
- key: eks.amazonaws.com/instance-category
operator: In
values: ["c", "m", "r"]
limits:
cpu: "1000"
memory: 1000Gi
disruption:
consolidationPolicy: Balanced
consolidateAfter: 1m
The NodeClass is where you express AWS infrastructure placement and node-level defaults. The NodePool is where you express what kind of capacity is acceptable for workloads. Do not copy a self-managed Karpenter EC2NodeClass into Auto Mode; Auto Mode uses its own NodeClass API.
A few defaults are worth knowing because they drive both disruption and cost. Auto Mode consolidates underutilized nodes, expires nodes after 336 hours (14 days) unless you change it, never lets a node live beyond 21 days, applies a disruption budget of 10% of nodes, and replaces nodes for drift when a new Auto Mode AMI ships, roughly once a week. consolidationPolicy accepts WhenEmpty, WhenEmptyOrUnderutilized and Balanced; the last one weighs disruption cost against savings and skips consolidations that are not worth it, which is a sensible default for most production pools. A billing label in the NodePool template (like billing-team above) is the cheapest way to get per-team cost allocation later.
Built-in components are managed differently than in a classic EKS build. Pod networking (with network policy enforcement), service networking, cluster DNS, compute autoscaling, EBS block storage, load balancing, the Pod Identity agent and node monitoring are Auto Mode capabilities. With Auto Mode compute, add-ons such as Amazon VPC CNI, kube-proxy, CoreDNS, the Amazon EBS CSI Driver and the EKS Pod Identity Agent become redundant for Auto Mode nodes, and the controllers run on AWS-owned infrastructure rather than as visible pods in your account. Autoscaling of replicas is still yours: Auto Mode adds nodes for pending pods, but you still need an HPA or KEDA to create those pods (see HPA best practices).
The workload support matrix is broader than Fargate, but not identical to self-managed nodes:
Capability
Auto Mode status
EC2 Spot
Supported through karpenter.sh/capacity-type (spot, on-demand, reserved)
Graviton / arm64
Supported through kubernetes.io/arch: arm64
GPU / accelerators
Supported for documented NVIDIA, Trainium and Inferentia families; drivers and device plugins managed
Capacity reservations
ODCRs and Capacity Blocks for ML via capacityReservationSelectorTerms
Windows nodes
Not supported (Linux only)
DaemonSets
Supported, but host-level assumptions must be validated against locked-down nodes
What changed in 2025 and 2026
Auto Mode has moved fast since launch, and several of the early “reasons not to use it” are gone. The ones worth knowing, all from AWS announcements:
October 2025, networking and security controls. The NodeClass gained podSubnetSelectorTerms and podSecurityGroupSelectorTerms for separate pod subnets, advancedNetworking options for public IP control and forward HTTPS proxies (httpsProxy, noProxy), certificateBundles for enterprise CAs, KMS encryption of root and ephemeral volumes, and support for On-Demand Capacity Reservations and Capacity Blocks for ML. Corporate proxies and private CAs used to be hard blockers; they are not anymore.
October 2025, GovCloud. Auto Mode became available in AWS GovCloud (US-East) and (US-West), and later in the AWS China Regions.
February 2026, enhanced logging. Auto Mode’s managed capabilities (compute autoscaling, block storage, load balancing, pod networking) can be configured as CloudWatch Vended Logs sources and delivered to CloudWatch Logs, S3 or Firehose. This closes the “black box” gap: you can finally see why the managed Karpenter did or did not launch a node.
April 2026, hidden managed instances. New Auto Mode instances and their resources are hidden by default from EC2 console views and API list operations. Good for accidental-deletion safety, surprising for cost and inventory tooling that lists EC2 instances directly. Adjust the managed resource visibility settings if your FinOps scripts rely on them.
July 2026, cheaper accelerated compute. Management fees cut by 35% for G-series and 60% for P-series and Trainium, effective July 1.
July 2026, zonal shift. Integration with Application Recovery Controller zonal shift: during a zonal shift Auto Mode stops provisioning in the impaired AZ and pauses voluntary disruption, at no extra cost.
July 2026, EFA and placement groups for Auto Mode node pools, aimed at distributed training.
Static capacity NodePools. Setting replicas on a NodePool keeps a fixed number of nodes regardless of demand, useful for latency-sensitive inference. Once set it cannot be removed, and static pools are never consolidated, so set limits.nodes above replicas to leave headroom for node replacement.
What you gain
Reduced operational surface. Node group management, AMI lifecycle, Cluster Autoscaler tuning, Karpenter controller upgrades, and a chunk of add-on wiring move out of your day-to-day scope.
Better provisioning shape by default. Dynamic provisioning is usually a better fit than fixed node group shapes. You get nodes that more closely match actual pod requirements instead of trying to pre-plan a small set of instance types, and that is exactly the efficiency that has to pay for the fee.
Automatic node patching. AWS manages the node image and replacement flow. That reduces toil, but it also means your workloads need disruption policies that let AWS replace nodes safely.
Faster cluster bootstrapping. A new Auto Mode cluster gets to a usable production baseline faster than a hand-assembled EKS cluster with node groups, autoscaling, networking add-ons, storage drivers, and load balancer controllers.
Native Spot integration. Spot interruptions, rebalance recommendations, scheduled maintenance and instance health events are handled for you, with no Node Termination Handler or SQS queue. You still need workload-level interruption tolerance: replicas, budgets, graceful shutdown, and queue semantics where relevant.
What you give up
Node-level access. Auto Mode nodes are intentionally locked down. If your incident response process assumes SSH, SSM, manual package inspection, or ad hoc host changes, it needs to change. Ephemeral debug containers become your main tool; the patterns in debugging distroless containers apply almost verbatim.
Custom AMIs. AWS determines the operating system and AMI for Auto Mode managed instances; you cannot directly access the instance or install software on it. If your organization requires internally built, hardened, or certified AMIs, Auto Mode is likely blocked. (For how Bottlerocket compares to other container-optimized OSes, see Talos vs Bottlerocket vs Flatcar.)
Unrestricted host agents. Kubernetes DaemonSets are supported, but they are the sharp edge. Anything that assumes privileged host access, custom kernel modules, hostPath writes, or low-level runtime integration needs a proof of compatibility.
Less tuning surface. You give up direct control over kubelet flags, container runtime configuration, bootstrap scripts, and arbitrary node setup. That is the point of the product, but it is also the boundary.
Different cost visibility. Managed node groups make capacity easy to reason about because you chose it up front. Auto Mode changes capacity dynamically and, since April 2026, hides its instances from default EC2 listings, so cost control moves toward NodePool limits, billing labels, Cost Explorer and workload-level resource hygiene.
When NOT to use EKS Auto Mode
Auto Mode is a good default, but there are clear cases where it is the wrong answer:
You need Windows nodes. Auto Mode is Linux only.
You must own the node image supply chain: certified or hardened AMIs, CIS images built in-house, or specific kernel modules.
Your security or observability stack needs host access that the locked-down Bottlerocket nodes do not allow, and the vendor has no supported Auto Mode deployment.
Your compute is already heavily discounted and efficiently packed. With mostly Spot or Savings Plans and a well-tuned Karpenter, the fee is a large relative premium for work you have already automated.
You run singleton stateful workloads that cannot tolerate node replacement at least every 21 days. Auto Mode will rotate the node anyway; if the PDB blocks it, you are just postponing an incident.
You rely on an alternative CNI or networking setup that Auto Mode does not support.
You need tight kubelet or runtime tuning (custom eviction thresholds, runtime configuration, special sysctls on the host).
When to use EKS Auto Mode
Use Auto Mode if:
You run EKS on AWS and do not have a hard requirement to manage nodes yourself
You do not require custom AMIs or custom node bootstrap logic
You want to reduce the operations surface for node lifecycle management
You want Karpenter-like provisioning without operating Karpenter yourself
Your workloads are mostly stateless or disruption-tolerant
Your observability, security, and storage agents are compatible with Auto Mode
Stick with managed node groups if:
Your organization requires internally certified or hardened AMIs
You need specific kernel configuration, kubelet flags, bootstrap scripts, or host packages
You depend on privileged DaemonSets or host-level security tooling that Auto Mode cannot support
You are in a regulated environment where the node image supply chain must be owned internally
Your platform team already operates Karpenter well and values the extra control
Use Fargate if:
You specifically want per-pod compute isolation
Your workload does not need DaemonSets or host-level integrations
You accept Fargate’s scheduling, networking, storage, and observability constraints
You want to avoid managing EC2 capacity entirely for a narrow class of workloads
Migration from managed node groups
Migrating an existing cluster to Auto Mode is supported, but it is not a one-command operational migration. You must update the cluster IAM role permissions and trust policy, enable the compute, block storage, and load balancing capabilities together, and meet required add-on versions when those add-ons are installed. AWS does not support direct migration of EBS volumes from the standard EBS CSI provisioner to the Auto Mode one, of existing load balancers from AWS Load Balancer Controller to Auto Mode, or of clusters using alternative CNIs. A conservative path looks like this:
Enable Auto Mode on a non-production cluster running a supported Kubernetes version (1.29 or later).
Inventory workloads by scheduling assumptions: node selectors, affinities, tolerations, topology spread, PDBs, privileged mode, hostPath, local storage, and DaemonSet dependencies.
Create or select the relevant Auto Mode NodeClass and NodePool resources.
Move a low-risk namespace first by changing selectors, tolerations, or labels so pods land on Auto Mode nodes (eks.amazonaws.com/compute-type: auto identifies them).
Taint old node groups to stop new scheduling once the pilot workloads are stable.
Drain old nodes gradually and delete old managed node groups only after workload owners have signed off.
You can keep AWS Load Balancer Controller installed during the migration when you need both models; separate them with IngressClass or loadBalancerClass and plan a blue-green move for existing load balancers.
The hardest part is usually not enabling Auto Mode. It is discovering which workloads and platform agents quietly depended on a mutable node.
What breaks when you migrate
Auto Mode changes the node contract. The Kubernetes API still looks familiar, but the host underneath is no longer yours in the same way.
DaemonSets need a compatibility audit. Logging agents, metrics agents, service mesh node components, security scanners, CSI node plugins, and custom infrastructure daemons often assume host access. Datadog, Falco, custom CSI drivers, eBPF agents, file integrity tools, and in-house node agents should be tested explicitly rather than assumed compatible.
PodDisruptionBudgets can block AWS-managed maintenance. If every critical Deployment has maxUnavailable: 0, or singleton workloads have no safe disruption path, node replacement becomes harder, and the 21-day lifetime means it will happen anyway. Auto Mode can manage nodes, but it cannot make an application disruption-tolerant after the fact.
nodeSelector and affinity rules can strand pods. Workloads pinned to old node group labels, instance types, capacity labels, zones, or custom AMI labels may never schedule on Auto Mode capacity. Replace legacy labels with stable requirements Auto Mode can satisfy, and remember that instance labels use the eks.amazonaws.com/ prefix, not karpenter.k8s.aws/.
topologySpreadConstraints can become too strict. Auto Mode provisions capacity dynamically, but strict zone spreading plus narrow selectors can create unschedulable pods. Check whenUnsatisfiable, label selectors, and minimum domain assumptions.
Privileged pods and hostPath volumes are migration blockers until proven otherwise. Anything that needs /var/lib, /proc, /sys, container runtime sockets, kernel capabilities, or host networking deserves a separate test. For a broader list of patterns that hurt under node churn, see Kubernetes features that hurt in production.
Observability and security agents may lose host assumptions. The dangerous failure mode is not CrashLoopBackOff; it is silent loss of telemetry or enforcement.
Storage drivers must be reviewed. EBS integration is part of Auto Mode, but it uses the provisioner ebs.csi.eks.amazonaws.com, not the standard ebs.csi.aws.com. Custom CSI drivers, EFS patterns, snapshot controllers, and topology-aware storage classes should be validated. Pay attention to provisioner names, volume binding mode, encryption settings, and IAM assumptions.
Runbooks need rewriting. “SSH to the node and inspect X” is not a valid first response anymore. Incident procedures should move toward kubectl describe, events, logs, ephemeral debug containers, the Auto Mode vended logs, cloud-side metrics, and vendor-supported diagnostics.
Cost and inventory scripts may go blind. Since April 2026, Auto Mode instances are hidden from default EC2 list operations. Anything that inventories or tags instances by listing EC2 directly needs the visibility setting changed or a switch to Kubernetes-side data.
1-week pilot: evaluate Auto Mode without risking production
Use a short pilot to answer compatibility and economics questions before touching production.
Day 1: build a representative test cluster. Same region, Kubernetes minor version, VPC shape, IAM model, ingress pattern, and storage classes where possible.
Day 1-2: enable Auto Mode with one custom NodeClass and NodePool. Keep the first pool boring: on-demand capacity, two or three common instance families, the same private subnet pattern as production, and a billing label.
Day 2-3: move three workload types. One stateless service, one stateful service with EBS, and one platform-heavy workload that uses observability or security agents. Run the scheduling audit (selectors, affinity, topology spread, PDBs, tolerations, privileged mode, hostPath, DaemonSets) before each move.
Day 4: force normal failure modes. Roll deployments, delete pods, scale replicas up and down, trigger consolidation, and simulate one Spot-tolerant workload if you plan to use Spot.
Day 5: validate platform signals. Confirm logs, metrics, traces, runtime alerts, security events, load balancer provisioning, DNS, and persistent volume operations. Enable the Auto Mode vended logs and check that you can explain every node launch.
Day 6-7: compare costs and toil. Record the EC2 instance mix, the published fee for each instance type in your region, pod density, pending time, interruption behavior, and operator actions required. Rerun the worked example above with your real numbers and your real discounts.
Success criteria should be explicit:
95%+ of pilot pods schedule without manual intervention
No silent loss of logs, metrics, traces, or security alerts
PDBs allow node replacement for replicated services
Stateful workloads survive rescheduling and volume attachment tests
Cost model is understood at instance-type level, including the fee on discounted capacity
Production migration blockers are documented with owners
If the pilot fails, that is still useful. It tells you which node assumptions are real and which workloads should stay on managed node groups.
The honest assessment
EKS Auto Mode is a good default candidate for many teams running Kubernetes on AWS, and a better one in late 2026 than at launch: proxies, private CAs, capacity reservations, logging and zonal shift were real gaps that have been closed. The operational simplification is real: node provisioning, AMI updates, core add-on integration, and scaling behavior are areas where teams burn time and create incidents.
The constraints are also real. Custom AMIs, Windows, unrestricted host access from DaemonSets, privileged pods, custom CSI drivers, and strict disruption policies are the common blockers. And the fee is not trivial for teams that already buy compute cheaply: it does not shrink with Spot or Savings Plans.
The right question is not whether Auto Mode is “better” than managed node groups. It is which operational contract your workloads can live with, and whether the fee is smaller than the node-management work it removes. The more your organization can treat nodes as replaceable capacity, the more value you get. The more your platform treats nodes as customized machines, the less Auto Mode fits.
Frequently Asked Questions
How much does EKS Auto Mode cost?
You pay the normal EKS control plane fee ($0.10 per cluster-hour on standard support), the normal EC2 price for the instances, and an EKS Auto Mode management fee per instance-hour that depends on instance type and region, billed per second with a one-minute minimum. In AWS’s published US West (Oregon) examples the fee equals 12% of the On-Demand price (for example $0.04128/hour on an m5a.2xlarge), but always check the rate for your instance types and region on the EKS pricing page.
Do Savings Plans, Reserved Instances or Spot reduce the EKS Auto Mode fee?
No. You can use On-Demand, Spot, Reserved Instances and Compute Savings Plans for the EC2 capacity that Auto Mode launches, and those discounts apply to the EC2 part of the bill, but the Auto Mode management fee is independent of the purchase option. That is why the fee is a larger relative premium for teams that already run mostly Spot or discounted capacity.
Is EKS Auto Mode cheaper than managed node groups?
Not on the raw bill for the same capacity: managed node groups have no management fee. Auto Mode can still lower total cost if its consolidation and right-sized instance selection remove enough idle capacity (around 11-12% better packing breaks even at On-Demand prices) or if it saves meaningful platform engineering time. Measure it with a pilot and your real discounts.
What is the difference between EKS Auto Mode and Karpenter?
Auto Mode runs Karpenter for you. Both use the same karpenter.sh/v1 NodePool API, but Auto Mode uses its own NodeClass (eks.amazonaws.com/v1) instead of EC2NodeClass, a locked-down Bottlerocket AMI chosen by AWS, native Spot interruption handling and managed GPU drivers. Self-managed Karpenter has no fee and lets you choose AMIs, OS and node lifetime, but you operate the controller, IAM, interruption queue and upgrades.
Can I enable EKS Auto Mode on an existing cluster?
Yes. Auto Mode can be enabled on existing clusters on a supported Kubernetes version, provided the cluster meets the IAM, add-on and networking requirements. Existing managed node groups keep running while you move workloads gradually. EBS volumes from the standard CSI provisioner and existing AWS Load Balancer Controller load balancers do not migrate directly and need a planned cut-over.
Can I use my existing Karpenter NodePools and EC2NodeClasses with Auto Mode?
Not directly. NodePools are the same API and usually port with small changes (instance labels use the eks.amazonaws.com/ prefix), but the self-managed EC2NodeClass has no equivalent in Auto Mode; you need to recreate its settings as an Auto Mode NodeClass and review every field before porting anything.
Does EKS Auto Mode support Windows nodes or SSH access?
No to both. Auto Mode supports only Linux nodes, and the nodes do not allow SSH or SSM access. Windows workloads should stay on managed node groups or self-managed nodes, and troubleshooting moves to Kubernetes-level tools such as ephemeral debug containers, events, logs and the Auto Mode vended logs.
Does EKS Auto Mode replace HPA or KEDA?
No. Auto Mode handles node provisioning and lifecycle: it adds nodes when pods are pending and removes them when they are not needed. It does not decide how many replicas your application should run. You still need HPA, KEDA or application-level scaling for pod replica counts.
The Problem: Docker Is Overkill for Image Operations
You need to copy an image from Docker Hub to your private registry. Or inspect a manifest before pulling. Or delete old tags programmatically. Or sync an entire repository during a migration.
The instinct is to reach for Docker. But Docker requires a running daemon, root access (or group membership that amounts to the same thing), and pulls the entire image to disk just to read its metadata. For CI pipelines, GitOps workflows, and platform tooling, that’s a significant overhead for what should be lightweight registry operations.
This is the problem that daemonless container image tools solve. Skopeo pioneered the category; today it has real competition from crane, regctl, and ORAS — each with different strengths and ideal use cases.
This article gives you the practical comparison to pick the right tool for your workflow.
The Contenders
Tool
Maintainer
Language
Daemon required
Skopeo
Red Hat / containers
Go
No
crane
Google / ko-build
Go
No
regctl
regclient
Go
No
ORAS
CNCF
Go
No
cosign
Sigstore / OpenSSF
Go
No
All five are Go binaries, statically compiled, and work directly against the OCI Distribution Spec. None of them require Docker or any container runtime.
Skopeo
Skopeo was the first major tool to address daemonless image operations, released by Red Hat in 2016 as part of the containers/image ecosystem (alongside Podman and Buildah).
What Skopeo does well
Image inspection without pulling:
skopeo inspect docker://registry.k8s.io/pause:3.9
Returns full image metadata — digest, layers, labels, architecture, OS — without downloading a single layer. Useful in admission controllers, policy checks, and pre-deployment validation.
The --all flag copies the full manifest list, preserving all architectures (linux/amd64, linux/arm64, etc.). This is critical when mirroring images for multi-arch clusters.
skopeo sync mirrors an entire repository, including all tags. You can also use a YAML file to define which images and tags to sync — useful for air-gapped environment bootstrapping.
Useful in CI for cleanup pipelines. Note that registry-side deletion requires the registry to have the DELETE method enabled.
Skopeo’s weaknesses
No image modification: Skopeo copies and inspects, but doesn’t build or modify images
Tag listing is verbose: skopeo list-tags returns JSON you need to parse
No retry logic by default: transient network errors in long sync operations require wrapping with retry scripts
Auth configuration: relies on containers/auth.json format, which differs from Docker’s ~/.docker/config.json (though it supports both)
When to use Skopeo
Air-gapped environment image mirroring
CI pipelines that need to copy or inspect images without Docker
Platform teams on Red Hat / OpenShift stacks
Any workflow already using Podman or Buildah
Crane
Crane is Google’s answer to Skopeo, developed as part of the ko project and later extracted into its own tool. It’s simpler, more scriptable, and has a cleaner CLI design.
What Crane does well
Tag listing:
crane ls registry.k8s.io/pause
No JSON parsing needed. One tag per line. Pipe directly into grep, sort, head.
Digest resolution:
crane digest docker.io/library/nginx:1.27
Returns the image digest. Combine with yq or sed to pin image references in Helm values or Kubernetes manifests.
Squashes all layers into one. Reduces layer count for images where layer history doesn’t matter.
Crane’s weaknesses
No sync command: unlike Skopeo, crane has no built-in repository sync. You script it yourself with crane ls + crane cp in a loop
Less mature multi-arch support: crane cp supports multi-arch but the UX is less explicit than Skopeo’s --all
No delete command: doesn’t implement registry deletion
When to use Crane
CI/CD scripting where you want clean, pipeable output
Digest pinning workflows
Lightweight image tagging operations
When you’re already in the ko / Google Cloud ecosystem
regctl
Regctl is the least known of the three but arguably the most feature-complete. It’s the CLI for the regclient Go library and covers use cases that Skopeo and crane leave out.
What regctl does uniquely well
Image modification without rebuild:
regctl image mod myimage:tag
--label "org.opencontainers.image.version=1.2.3"
--replace
You can add/change labels, annotations, and config fields directly on an existing image in the registry — without pulling, rebuilding, or pushing a new image. This is impossible with Skopeo or crane.
Layer operations:
# Remove a specific layer from an image
regctl image mod myimage:tag
--layer-rm sha256:abc123...
--replace
Useful for removing accidentally included secrets or large unnecessary layers from published images.
OCI artifact support:
regctl artifact put
--media-type application/vnd.example.config.v1+json
--config config.json
file.tar.gz
harbor.internal/myartifacts:v1
Regctl has solid OCI artifact support alongside standard image operations.
Formatting and output:
regctl tag list harbor.internal/myapp --format '{{range .}}{{println .}}{{end}}'
Go template formatting throughout. Useful for integrating into shell scripts without jq.
Lists referrers (signatures, SBOMs, attestations) attached to an image via the OCI 1.1 referrers API.
Regctl’s weaknesses
Smaller community: fewer examples, less StackOverflow coverage
Steeper learning curve: more commands, more flags
Less packaging: not in most distro repos by default
When to use regctl
Image post-processing (labels, annotations, layer removal) without rebuild
Advanced manifest and referrer workflows
When you need OCI artifact operations alongside image operations
ORAS
ORAS (OCI Registry As Storage) is a CNCF project focused specifically on OCI artifact management — pushing and pulling arbitrary files to container registries, not necessarily container images.
# Push a Helm chart as an OCI artifact
oras push harbor.internal/charts/myapp:1.0.0
--artifact-type application/vnd.helm.chart.v1+tar
mychart.tgz
# Push SBOM
oras push harbor.internal/myapp:v1
--artifact-type application/spdx+json
sbom.spdx.json
# Pull
oras pull harbor.internal/charts/myapp:1.0.0
ORAS is not a direct Skopeo replacement — it’s for when your registry is a general-purpose artifact store, not just a container registry. Helm OCI, SBOMs, attestations, and policy bundles all benefit from ORAS.
ORAS vs Skopeo in one line: Skopeo (and crane) move container images between registries; ORAS pushes and pulls arbitrary artifacts (charts, SBOMs, ML models) to a registry. If your object is an OCI image, use Skopeo or crane. If it is a file you want to store in a registry, use ORAS. They overlap far less than the shared “registry client” label suggests, and most platform teams end up with both installed for different jobs.
cosign
Cosign from Sigstore is not a general-purpose image tool — it’s specifically for supply chain security. But it’s increasingly part of any container image workflow.
Cosign integrates with OIDC providers for keyless signing (no key management required), which is the direction the ecosystem is moving. If you’re building a supply chain security practice, cosign is mandatory, not optional.
Side-by-side comparison
Operation
Skopeo
Crane
regctl
Inspect image
inspect
manifest
manifest get
Copy image
copy
cp
image copy
Copy all arches
copy --all
cp (auto)
image copy
Sync repository
sync
script it
script it
List tags
list-tags (JSON)
ls (plain)
tag list
Delete tag/image
delete
—
tag delete
Modify labels
—
—
image mod
Remove layer
—
—
image mod
OCI artifacts
limited
limited
artifact
Referrers (1.1)
—
—
manifest get-list
Practical workflows
Copy an image from Docker Hub to a private registry (crane cp vs skopeo copy)
This is the single most common reason people reach for these tools, so here are the two one-liners side by side. Both copy the image directly registry-to-registry — the image never touches your local disk and no Docker daemon is involved.
# skopeo copy — explicit docker:// transport on both sides
skopeo copy
docker://docker.io/library/nginx:1.27
docker://harbor.internal/library/nginx:1.27
# crane cp — shorter, no transport prefix
crane cp docker.io/library/nginx:1.27 harbor.internal/library/nginx:1.27
They are functionally equivalent for a single image. Reach for skopeo copy when you want --all to force the full multi-arch manifest list, or when you are on a Red Hat / Podman stack. Reach for crane cp when you want the shortest possible command and cleaner output for scripting — it copies the manifest list automatically when the source is multi-arch. To authenticate against the private registry first, both read ~/.docker/config.json, so a prior docker login harbor.internal (or crane auth login / skopeo login) is enough. To copy many images or whole repositories in one go, jump to the air-gapped mirroring recipe below, which uses skopeo sync.
#!/bin/bash
# Update image digests in values.yaml
for image in nginx:1.27 redis:7.4; do
digest=$(crane digest docker.io/library/${image})
echo "docker.io/library/${image}@${digest}"
done
Combine with yq to update Helm values files automatically, ensuring reproducible deployments.
Retag without re-pushing (Crane or regctl)
# After a successful deploy to staging, promote to production
crane tag harbor.internal/myapp:${GIT_SHA} harbor.internal/myapp:production
No layer transfer. The operation is a metadata update in the registry.
Add OCI annotations post-build (regctl)
regctl image mod harbor.internal/myapp:v1.2.3
--annotation "org.opencontainers.image.source=https://github.com/org/repo"
--annotation "org.opencontainers.image.revision=${GIT_SHA}"
--replace
Attaches build metadata to an image already in the registry, without a rebuild.
Use Skopeo if: you’re on a Red Hat / OpenShift stack, you need repository sync, or you’re building air-gapped environment pipelines. It’s the most battle-tested and widely packaged.
Use Crane if: you’re scripting image operations in CI and want clean, composable CLI output. crane ls + crane cp + crane digest cover 80% of automation use cases with minimal friction.
Use regctl if: you need to modify images post-build, work with OCI referrers, or want the most complete feature set for a registry client. It has a higher learning curve but can replace both Skopeo and crane for advanced workflows.
Use ORAS if: you’re using a registry to store non-image artifacts — Helm charts, SBOMs, policy bundles, ML models.
Use cosign regardless of which of the above you pick, as soon as supply chain security matters to your organization. It’s not a replacement for the others — it’s a complement.
In practice, most platform teams end up using 2-3 of these tools together. Crane for day-to-day scripting, Skopeo for sync jobs, cosign for signing, ORAS for artifact storage.
FAQ
Can I use these tools with private registries?
Yes. All support standard registry authentication. Crane and Skopeo both read from ~/.docker/config.json. Regctl has its own config file (~/.regctl/config.json) but can import Docker credentials. Set DOCKER_CONFIG to point to your credentials file in CI environments.
Do these work with Docker Hub rate limits?
Yes, and they’re often more efficient than the Docker CLI because they only fetch manifest metadata for inspect operations, not full layers. For heavy pull workloads, authenticate with your Docker Hub credentials to get higher rate limits.
What about ECR, GCR, and Azure Container Registry?
All tools support these with the appropriate credential helpers. For ECR, use docker-credential-ecr-login. Crane has native ECR support via the --platform flag and crane auth commands. Skopeo supports ECR via --creds "AWS:$(aws ecr get-login-password)".
Are these tools safe to run in Kubernetes pods?
Yes. Since they require no daemon and no elevated privileges for read operations, they’re well-suited to run as init containers or sidecar containers in Kubernetes. Skopeo is commonly used in image pre-pulling init containers. Use a dedicated service account with least-privilege registry credentials.
Can I copy a multi-arch image and keep all platforms?
Skopeo: skopeo copy --all. Crane: crane cp copies the index automatically when the source is a manifest list. Regctl: regctl image copy preserves manifest lists by default.
If you’ve been following Kubernetes tooling, you’ve probably already been through the Lens saga: Lens went commercial, OpenLens emerged as the community fork, then FreeLens appeared when OpenLens maintenance slowed. The pattern is familiar — a useful desktop tool, a licensing decision, a fork, another fork. Radar is not a fork. It’s a different approach to the same problem: giving engineers a useful interface for Kubernetes clusters without the friction of kubectl for every task. Built by Skyhook (YC-backed, Google Cloud Partner), it’s been live since 2025, has 1.7k+ GitHub stars, releases weekly, and the founder reaches out to the community directly. That’s usually a good signal that someone is genuinely building in public. This article covers what Radar actually does, where it pulls ahead of OpenLens and FreeLens, and when those tools are still the right choice.
The State of Kubernetes Desktop Tooling in 2026
Before getting into Radar specifically, it’s worth naming the landscape clearly:
Lens — the original. Electron-based, polished, now commercial (Mirantis). The free Personal tier is non-commercial only. Pro is ~$22-35/user/month.
OpenLens — the community fork of Lens before Mirantis closed exec/logs/shell in v6.3 (January 2023). Maintenance has slowed significantly. No active release cadence.
FreeLens — a more active community fork, filling the gap left by OpenLens’ decline. Restores the missing features. No commercial backing.
k9s — terminal TUI, fast, keyboard-driven, single-cluster. Different audience.
Radar — Go binary, Apache 2.0, team-oriented, topology and event timeline focused.
The problem with OpenLens and FreeLens is not that they’re bad tools — they’re genuinely useful for the solo developer with one or two clusters. The problem is that they’re single-cluster-at-a-time desktop apps with no concept of team, no persistent state, and no awareness of the modern Kubernetes ecosystem (ArgoCD, Flux, Karpenter, KEDA). As your infrastructure grows, you outgrow them.
What Radar Actually Is
Radar is available in two forms:
Radar OSS — a single ~30MB Go binary, Apache 2.0, free forever. Can run locally (desktop app) or deployed in-cluster via Helm. No sidecars, no feature gates.
Radar Cloud — same binary, adds a hosted control plane with fleet aggregation and shared URLs for team incident response. Free for up to 3 clusters (24-hour timeline retention); Team is $149/cluster/month (scoped RBAC, 3-day retention) and Enterprise $299/cluster/month (SAML/OIDC SSO, SCIM, 30-day retention). Priced per cluster, not per user — users are unlimited, and annual plans get 20% off.
The per-cluster pricing is a deliberate design decision — teams don’t pay more as they add engineers, only as they add clusters. For a 20-person platform engineering team managing 5 clusters, Radar Cloud Team runs $745/month ($1,495/month on Enterprise if you need SSO/SCIM). The equivalent Lens Pro seats would cost $2,200-4,200/month. For most self-hosted environments, the OSS version is sufficient and costs nothing.Radar’s home view on a 3-node Talos cluster: workload counts, CPU/memory requests, Helm releases and active issues in one screen.
Key Features
Topology View
This is the most visually distinctive feature. Radar renders a live service graph for your cluster: deployments, services, ingresses, cross-namespace dependencies, and east-west traffic flows — all in a single view without running kubectl get all -A and stitching the output together mentally. OpenLens and FreeLens have resource list views. They show you what exists. Radar shows you how things connect — which is what you actually need when debugging why Service A can’t reach Service B.Topology view of the observability namespace (kube-prometheus-stack, Loki, Grafana): ingress → service → workload → pod.
Persistent Event Timeline
Kubernetes events are ephemeral by default — they expire after approximately one hour. When something breaks at 2am and you’re looking at it at 9am, the events that explain what happened are gone. Logs may still be there if you’re running a log aggregator, but the Kubernetes-level events (pod restarts, scheduling failures, node pressure events, probe failures) are gone. Radar retains events. The OSS version extends this beyond the default 1-hour cluster retention. Radar Cloud retains up to 30 days on the Enterprise tier (3 days on Team). You can rewind the timeline to any point and reconstruct what the cluster looked like at that moment. Neither OpenLens nor FreeLens have any event retention beyond what the cluster itself provides.The event timeline groups changes per resource and app, so you can scroll back instead of racing the 1-hour event TTL.
GitOps Integration (ArgoCD + Flux)
Radar auto-detects ArgoCD and Flux and surfaces sync state, drift, and health directly in the UI. You can see whether a deployment is in sync, when it last synced, and whether it drifted from the desired state in Git. In OpenLens and FreeLens, ArgoCD resources appear as generic Kubernetes custom resources. You can see the CRDs, but there’s no purpose-built understanding of what they mean — no sync status visualization, no diff view, no rollback trigger.GitOps view: every ArgoCD Application with sync state, health, source and last sync — the degraded and out-of-sync ones surface at the top.
Helm Management
Radar tracks Helm releases with full revision history and supports one-click rollbacks from the UI. This is similar to what OpenLens/FreeLens offer via the Helm releases view, but Radar adds revision diffing — you can see what changed between release 5 and release 6 before deciding to roll back.
Image Filesystem
You can browse container image filesystems through Radar without needing kubectl exec into a running pod or access to the container registry. Useful for security audits and debugging — you can verify what’s actually in an image at rest.
MCP Server (AI Integration)
Radar ships with an MCP (Model Context Protocol) server, which means you can connect Claude, Cursor, or GitHub Copilot directly to your cluster context and ask questions about it in natural language. The MCP server is token-optimized — it doesn’t dump raw YAML at the model, it structures cluster state into meaningful context. This is something neither OpenLens nor FreeLens have. It’s also something that’s genuinely useful if you’re already using AI assistants for development work.
Cluster Audit
30 built-in best-practice checks — resource requests/limits, RBAC permissions, image pinning, network policies, security contexts. The checks are labeled by compliance framework. This is not a replacement for dedicated security tooling (Trivy, Falco, Polaris), but it’s a useful first-pass audit without leaving the tool you’re already using.
Multi-Cluster Support (Cloud)
The Cloud tier adds fleet-level visibility: a single view across all clusters, cross-cluster search, and drift detection between environments (e.g., staging vs. production). This is the feature that changes the calculus for platform engineering teams managing 5+ clusters. OpenLens and FreeLens require you to switch cluster context manually. There is no fleet view.
Architecture: Why a Go Binary Matters
OpenLens and FreeLens are Electron apps — Chromium + Node.js wrapped in a desktop shell. This means:
200-500MB install size
1-2 second startup time on a fast machine, more on slower ones
Memory footprint in the hundreds of megabytes
Local kubeconfig required on each engineer’s machine
Radar’s in-cluster deployment is a single Go binary (~30MB) that runs as a Pod with a ServiceAccount. It connects to the hosted control plane over outbound WebSocket + TLS. No inbound firewall rules, no kubeconfig distribution, no per-engineer setup. The local desktop app is also a lightweight Go binary — 65-second startup was demonstrated on a 322-node cluster. That’s not a typo. For in-cluster deployment, the architecture means security is handled at the ServiceAccount level, not by distributing kubeconfigs to engineer laptops. That matters for teams with security requirements around credential management.
Feature Comparison
Feature
Radar OSS
Radar Cloud
OpenLens
FreeLens
License
Apache 2.0
Proprietary (hosted)
MIT/GPL
MIT
Maintenance
Active (weekly releases)
Active
Stalled
Active (community)
Architecture
Go binary / in-cluster
In-cluster + hosted
Electron
Electron
Multi-cluster
Basic
Fleet view
❌
❌
Event retention
Extended
24h free / 3 days Team / 30 days Enterprise
Cluster default (~1h)
Cluster default (~1h)
Topology view
✅
✅
❌
❌
GitOps (ArgoCD/Flux)
✅
✅
CRDs only
CRDs only
Helm management
✅
✅
✅
✅
kubectl exec / logs / shell
✅
✅
✅ (restored)
✅
MCP / AI integration
✅
✅
❌
❌
Cluster audit
✅
✅
❌
❌
SSO / SCIM
❌
✅
❌
❌
Shared incident URLs
❌
✅
❌
❌
Image filesystem browser
✅
✅
❌
❌
Cost tracking
✅ (OpenCost)
✅
❌
❌
Price
Free
Free up to 3 clusters; from $149/cluster/month
Free
Free
When Radar Makes Sense
You’re managing multiple clusters. Even with the OSS version, the topology view and event timeline make Radar more useful than OpenLens/FreeLens at 3+ clusters. The Cloud fleet view is the compelling option at 5+. Your team uses GitOps. If ArgoCD or Flux is part of your workflow, Radar’s native understanding of sync state and drift is meaningfully better than seeing CRDs in a generic list view. You need post-mortem capability. If your incident review process involves looking at what the cluster was doing when the alert fired, you need event retention. Radar has it; OpenLens and FreeLens don’t. You’re adopting AI tooling. The MCP server is the most forward-looking feature here. If you use Claude Code, Cursor, or Copilot for your infrastructure work, having cluster context available to those tools without copy-pasting YAML is a genuine productivity improvement. You have a platform engineering team. Per-cluster pricing, SSO, SCIM, and shared incident URLs are features that only matter if you have more than one person managing infrastructure.
When OpenLens or FreeLens Still Makes Sense
You’re a solo developer with one or two clusters. OpenLens and FreeLens are familiar, local, and have zero setup overhead. If you don’t need team features, event retention, or topology views, they remain perfectly functional tools. You’re deeply invested in the Lens UX. The resource tree, the terminal integration, the way Lens presents namespace-scoped resources — if your muscle memory is built around that interface, switching has a real cost. Radar is different, not just better. You need maximum customization. OpenLens and FreeLens support plugins. Radar does not currently have a plugin system. Your environment is air-gapped or has strict egress restrictions. Radar OSS can run fully in-cluster, but Radar Cloud requires outbound connectivity to the hosted control plane. OpenLens and FreeLens are fully local.
Radar is the most interesting new entrant in the Kubernetes tooling space in a while — not because it replaces everything else, but because it addresses the specific gap that OpenLens and FreeLens never covered: teams, multiple clusters, and persistent state. For a solo developer, OpenLens or FreeLens are still completely reasonable choices. For a platform engineering team managing more than two clusters with ArgoCD or Flux, Radar’s feature set is materially better and the OSS version costs nothing. The active release cadence and the YC backing suggest this isn’t a one-person side project — there’s a team actively working on it. Whether the Cloud pricing sticks long-term is a question only usage will answer, but the Apache 2.0 core with an explicit “always open source” commitment is the right foundation. Worth evaluating if you haven’t already.Tested with Radar OSS v0.x on Kubernetes 1.29–1.32. Feature availability as of May 2026; Radar Cloud pricing updated October 2026. Screenshots: Radar v1.16 on a 3-node Talos Linux cluster (Kubernetes 1.37), October 2026.
