Talos Linux vs Bottlerocket vs Flatcar Container Linux: choosing an immutable OS for Kubernetes nodes (2026)

Talos Linux vs Bottlerocket vs Flatcar Container Linux: choosing an immutable OS for Kubernetes nodes (2026)

Kubernetes node operating systems have become a real design choice again. For years, most teams ran Ubuntu, Debian, RHEL, or Amazon Linux on workers, then layered kubelet, containerd, hardening, patching, observability agents, GPU drivers, and configuration management on top. That works, but it leaves a large mutable host underneath a scheduler that treats nodes as replaceable capacity.

Immutable container-focused operating systems attack the problem differently. They reduce the host to a small, versioned artifact, make updates atomic, and move configuration into declarative APIs or boot-time documents. The goal is fewer moving parts, less drift, smaller attack surface, predictable upgrades, and replaceable nodes.

In 2026, the three names that come up most often are Talos Linux, Bottlerocket, and Flatcar Container Linux. They overlap, but they are not interchangeable:

  • Talos Linux is the most Kubernetes-specific and the most radical: no SSH, no shell, no package manager, API-only operations.
  • Bottlerocket is the AWS-native container host: strong fit for EKS and ECS, image variants, API settings, admin containers, and AWS support.
  • Flatcar Container Linux is the evolutionary successor to the CoreOS Container Linux model: read-only system partition, Ignition/Container Linux Config, systemd, SSH, automatic updates, and broad platform coverage.

For a step-by-step Talos implementation guide, read the deep dive: Talos Linux: The Immutable, API-Driven OS for Kubernetes. This article is a decision guide: what changes operationally, where each OS fits, and what trade-offs senior platform teams should care about.

The decision in one paragraph

Choose Bottlerocket when your platform is AWS-based and you want a supported node OS that integrates with EKS, EC2, ECS, AMIs, and AWS support plans. Choose Talos when you want a Kubernetes-first OS across bare metal, edge, private cloud, or multi-cloud, and can replace SSH-era habits with an API-driven model. Choose Flatcar when you want immutable infrastructure and automatic updates, but still need systemd, SSH, Ignition-style provisioning, and a smoother migration from Ubuntu or CoreOS-like patterns.

What Talos Linux is

Talos Linux is a Kubernetes-optimized Linux distribution from Sidero Labs. Its design is deliberately narrow: it exists to run Kubernetes. It is not a general-purpose server distribution with a hardened profile. The host has no SSH daemon, no interactive shell, and no package manager. Configuration and day-2 operations go through talosctl and an authenticated gRPC API using mutual TLS.

That changes the security model. On a traditional node, operational access usually means SSH, sudo, shell history, mutable filesystems, ad hoc commands, and sometimes a long tail of packages that were installed to debug one incident and never removed. Talos removes that path. You inspect resources, logs, services, disks, and Kubernetes component state through the Talos API. You change machine configuration by applying declarative config patches. You upgrade by asking the node to move to a new installer image.

The upside is a very small node surface. Sidero states that Talos ships with fewer than 50 OS binaries and no package manager; current releases also publish SBOM, VEX, and cryptographic signatures. Talos upgrades use an A/B image scheme so the previous kernel and OS image remain available for rollback. As of this review, the latest GitHub release is Talos v1.13.5, published on June 22, 2026.

The cost is operational discipline. If your incident response muscle memory is ssh, strace, edit a file, restart a daemon, Talos forces a redesign around API resources, Kubernetes-level observability, ephemeral debug containers where appropriate, and reproducible configuration.

What Bottlerocket is

Bottlerocket is an open-source Linux-based operating system purpose-built by Amazon Web Services for running containers. AWS provides Bottlerocket as no-cost AMIs for EC2 and documents it as a container host for EKS, ECS, VMware, and bare metal. The product center of gravity is clearly AWS.

Bottlerocket does not use a general-purpose package manager. Instead, it ships as variants: predefined images tailored to platform, architecture, orchestrator, orchestrator version, and sometimes flavor. Kubernetes variants are tied to Kubernetes minor versions, and Bottlerocket also has NVIDIA and FIPS flavors. The Bottlerocket OS version is separate from the Kubernetes variant version. As of this review, the current docs are under the 1.62.x branch and the latest GitHub release is v1.62.1, published on June 22, 2026.

Configuration is modeled and exposed through an API. Updates are image-based and use partition flips rather than package mutation. For Kubernetes clusters, the Bottlerocket Update Operator, or Brupop, coordinates in-place node updates. It runs a controller, node agents, and an API server, drains nodes through the eviction API, respects PodDisruptionBudgets, and rolls updates in waves.

Bottlerocket is less austere than Talos. It supports host and admin container concepts for controlled access, which helps with break-glass paths, SSM-style workflows, and AWS support expectations. The trade-off is portability: it can run outside EC2, but its strongest ecosystem, release consumption model, and support story are AWS-shaped.

What Flatcar Container Linux is

Flatcar Container Linux is a community Linux distribution for container workloads. It started as a fork of CoreOS Container Linux and now lives in the Cloud Native Computing Foundation. CNCF lists Flatcar as an Incubating project, accepted on August 2, 2024.

Flatcar is immutable in the classic Container Linux sense. The system partition is read-only, the OS is delivered as an image, updates are automatic and atomic, and the node is provisioned declaratively at boot. Unlike Talos, Flatcar keeps a familiar Linux model: systemd, SSH, files, units, and host-level customization. The current recommended provisioning path is Container Linux Config transpiled to Ignition. The older cloud-config model remains useful historically, but the docs recommend Container Linux Config for new provisioning.

Flatcar’s platform coverage is broad, including major clouds, OpenStack, Proxmox, VMware, KubeVirt, libvirt, QEMU, and VirtualBox. The release page currently shows Stable 4593.2.3, Beta 4694.1.0, Alpha 4722.0.0, and LTS 4081.3.8. The LTS model is useful for teams that want fewer feature changes.

Flatcar’s day-2 profile is closest to traditional Linux, which is both its strength and its risk. You can carry forward systemd-based automation, SSH debugging, and existing tools. You can also carry forward drift if you treat it like a mutable server.

Comparative table

AreaTalos LinuxBottlerocketFlatcar Container Linux
Core philosophyKubernetes-only, API-managed, no shellAWS-built container host with variantsContainer Linux successor with systemd and Ignition
Best fitBare metal, edge, private cloud, multi-cloud, high-security clustersEKS/ECS and AWS-heavy fleetsUbuntu/CoreOS-style migrations that still need Linux primitives
Management modeltalosctl over mTLS gRPC; declarative machine configAPI settings, variants, AMIs, BrupopContainer Linux Config/Ignition, systemd, SSH, channels, Nebraska
SSH and shellNo SSH, no shellNot a normal SSH-first OS; controlled access through host/admin containersSSH and normal Linux debugging patterns are available
Package managerNoneNone; use variants insteadNo traditional package workflow for the base OS; extend through containers, Ignition, systemd, and systemd-sysext
UpdatesAPI-triggered upgrades, A/B image scheme, rollback pathPartition-flip updates; Brupop for Kubernetes nodesAutomatic atomic updates via channels and reboot strategies
Kubernetes couplingVery high; control plane and workers are first-classHigh for Kubernetes variants; also ECSMedium; Kubernetes lifecycle is assembled above the OS
Attack surfaceSmallest and most opinionated; no SSH, no shell, no package managerSmall, container-focused, no package manager, AWS security postureMinimal container OS, read-only system partition, but broader Linux surface than Talos
Platform supportBroad: bare metal, VMs, cloud, edgeStrongest on AWS; also VMware and bare metal variantsBroad cloud and virtualization support
ExtensionsSystem extensions, Image Factory, extension services, release-bound GPU supportVariants and flavors such as NVIDIA and FIPSsystemd-sysext, containers, systemd units, sysext-bakery
Governance/companyOpen-source Talos by Sidero Labs; Sidero is a CNCF Silver MemberOpen-source project built by AWS; AWS supports AWS-provided buildsCNCF Incubating project since August 2024
Learning curveHighest if the team expects SSHModerate for AWS teams; lower inside EKS practicesLowest for Linux operations teams

Day-2 operations: upgrades, debugging, and extensions

The day-2 question matters more than installation. You will install the OS once per node lifecycle, but upgrade, debug, replace, patch, and explain incidents for years.

Talos is the most coherent if you want every node operation to be declarative and auditable. Upgrades are API calls against an installer image. Kubernetes and OS upgrades are explicit operations, not package side effects. Diagnostics come from talosctl: service status, logs, disks, network state, component health, and resource APIs. Runbooks need Talos resources, not shell commands.

Extensions in Talos are powerful but constrained. System extensions are included in generated boot media or installer images, commonly via Image Factory. Extension services can run early privileged services where static pods or DaemonSets are not enough. Hardware-specific work, such as NVIDIA support, is possible but release-bound.

Bottlerocket day-2 operations fit AWS lifecycle thinking. In EKS, Brupop handles in-place updates for existing Bottlerocket nodes. It drains nodes, uses the eviction API, respects PDBs, and limits rollout blast radius. New nodes still depend on the AMI or image you launch, so keep launch templates current.

Debugging Bottlerocket is less disruptive than Talos for teams that want controlled host access, because the OS supports admin and host containers. Treat that as break-glass access, not routine configuration.

Flatcar day-2 operations are the most familiar. Automatic updates arrive through channels. Production clusters usually use Stable; a subset can run Beta to catch issues earlier. Reboot behavior is controlled through documented strategies, and organizations can run their own Nebraska-style update infrastructure. Flatcar also has an LTS channel.

For extensions, Flatcar has become more modular through systemd-sysext: overlays on read-only /usr, prebuilt sysext-bakery images, Kubernetes binaries, CRI-O, K3s, Wasmtime, and cloud vendor tooling.

When to choose Talos

Choose Talos when Kubernetes is the product boundary of the node. If nothing should run on the host except Kubernetes machinery, Talos gives you the cleanest model. It is compelling for bare metal, edge clusters, private cloud, Proxmox/vSphere, and multi-cloud fleets where you want one operational abstraction.

Talos is also the strongest choice when security teams want a hard answer to SSH access, mutable root filesystems, package drift, and undocumented host changes. Pair it with Kubernetes security best practices and the node layer becomes easier to audit.

Do not choose Talos casually if your team depends on node-level manual debugging, package-installed host agents, or mixed workloads outside Kubernetes. Talos rewards teams that invest in runbooks, observability, automated replacement, and reproducible node configuration.

When to choose Bottlerocket

Choose Bottlerocket when most of your Kubernetes fleet runs on AWS, especially EKS, and your organization values AWS-native support paths. Bottlerocket aligns with EC2 images, EKS worker patterns, AWS-provided builds, and AWS support plans.

Bottlerocket also fits teams that want immutable nodes but cannot go all the way to Talos’s no-shell model. Admin and host containers provide controlled access, variants reduce ambiguity, and Brupop coordinates rollout.

The caveat is portability. Bottlerocket can run in non-AWS contexts, but the further you move from AWS, the more you should compare it against Talos and Flatcar on ecosystem, tooling, and support.

When to choose Flatcar

Choose Flatcar when you want a lower-risk transition from traditional Linux nodes to immutable infrastructure. It is strong for teams that already understand systemd, Ignition, CoreOS-style provisioning, SSH, update channels, and cloud images. It gives you a read-only base OS, automatic atomic updates, broad platform availability, and CNCF governance without rewriting every host-level assumption.

Flatcar is also useful when Kubernetes is not the only container workload, or when Kubernetes, standalone container hosts, and custom systemd-managed components coexist. It is less opinionated than Talos and less AWS-centered than Bottlerocket.

The risk is familiarity. If engineers SSH into nodes, add one-off files, and treat systemd units as snowflake configuration, the immutable base loses value. Manage Flatcar with boot-time config, versioned artifacts, controlled update channels, and replaceable nodes.

FAQ

Is Talos more secure than Bottlerocket or Flatcar?

It depends on what you mean by secure. Talos has the smallest operational attack surface because it removes SSH, shell access, and package management entirely. Bottlerocket and Flatcar also reduce attack surface compared with general-purpose distributions, but preserve different forms of host access and customization.

Can Bottlerocket run outside AWS?

Yes, but AWS remains its center of gravity. Official materials describe variants for Kubernetes worker nodes in EC2, VMware, and bare metal, and ECS variants for EC2. For non-AWS-first environments, compare platform support, image lifecycle, and support expectations against Talos and Flatcar.

Is Flatcar just old CoreOS?

No. Flatcar started as a fork of CoreOS Container Linux, but it is now an active CNCF Incubating project with its own release streams, platform support, update mechanisms, and systemd-sysext-based modularity.

Which one is easiest to migrate to from Ubuntu worker nodes?

Flatcar is usually easiest because it preserves systemd, SSH, and a recognizable Linux host model while adding immutable OS updates. Bottlerocket can be straightforward inside EKS. Talos is often the biggest shift because it changes how administrators access and debug nodes.

Final recommendation

For AWS-heavy EKS fleets, start with Bottlerocket unless portability or a no-shell requirement points elsewhere. For bare metal, edge, private cloud, and multi-cloud Kubernetes, put Talos at the top of the shortlist and validate debugging and extension workflows early. For teams modernizing from Ubuntu or legacy CoreOS-style infrastructure, Flatcar is the pragmatic bridge.

The bigger decision is whether your platform treats nodes as replaceable, versioned infrastructure. Once that is true, all three can work well. The best choice is the one whose failure modes match your team’s reality.

Sources

  • https://www.siderolabs.com/talos-linux
  • https://docs.siderolabs.com/talos/v1.13/overview/what-is-talos
  • https://docs.siderolabs.com/talos/v1.13/configure-your-talos-cluster/lifecycle-management/upgrading-talos
  • https://docs.siderolabs.com/talos/v1.13/build-and-extend-talos/custom-images-and-development/system-extensions
  • https://docs.siderolabs.com/talos/v1.13/build-and-extend-talos/custom-images-and-development/extension-services
  • https://docs.siderolabs.com/talos/v1.13/learn-more/image-factory
  • https://github.com/siderolabs/talos/releases
  • https://aws.amazon.com/bottlerocket/
  • https://aws.amazon.com/bottlerocket/faqs/
  • https://bottlerocket.dev/en/os/1.62.x/concepts/variants/
  • https://github.com/bottlerocket-os/bottlerocket
  • https://github.com/bottlerocket-os/bottlerocket/releases
  • https://github.com/bottlerocket-os/bottlerocket-update-operator
  • https://www.cncf.io/projects/flatcar-container-linux/
  • https://www.flatcar.org/
  • https://www.flatcar.org/releases
  • https://www.flatcar.org/docs/latest/provisioning/cl-config/from-cloud-config/
  • https://www.flatcar.org/docs/latest/setup/releases/update-strategies/
  • https://www.flatcar.org/blog/2024/04/os-innovation-with-systemd-sysext/

OpenSearch, Loki, Quickwit and ClickHouse: Which Platform Should You Choose for Logs and Observability in 2026?

OpenSearch, Loki, Quickwit and ClickHouse: Which Platform Should You Choose for Logs and Observability in 2026?

Log management and observability are no longer a matter of “storing text and searching it quickly.” In 2026, the real challenge is economic and architectural: ingesting growing volumes, retaining data for months, correlating logs with metrics and traces, and responding during an incident without costs spiraling out of control.

For years, Elasticsearch was the dominant option. Then came OpenSearch, Loki, Quickwit and, with significant momentum, ClickHouse as an analytical backend. All of them can solve part of the problem, but they do not optimize for the same things. The senior decision should not be “which one is faster” in the abstract, but which questions the team will ask, what retention it needs and how much complexity it can operate.

The Historical Standard: Elasticsearch and OpenSearch

OpenSearch began as an open source fork of Elasticsearch and keeps the mental model most familiar to many teams: indices, shards, replicas, mappings, Query DSL, aggregations and a mature full-text experience. If your organization already comes from Elasticsearch, it is the most direct transition.

Architecturally, it is built on Apache Lucene. Each index is divided into shards; each shard maintains segments and inverted indices; the cluster coordinates writes, replicas, distributed searches and aggregations. This provides a great deal of power, but also a clear operational footprint: JVM, heap sizing, merges, hot/warm/cold tiers and mapping tuning.

For logs, OpenSearch shines when you need expressive text search, rich filters, aggregations, mature dashboards, existing tooling or use cases close to SIEM. The cost appears as volume grows: indexing a lot of content makes it fast to search, but consumes storage and memory.

In 2026, the 3.x branch also matters. OpenSearch 3.0 introduced relevant changes such as Lucene 10 and JVM 21, along with performance improvements and vector search capabilities. That signals project vitality, but it also means upgrades need more careful planning than in a purely stateless stack.

Loki: Observability Centered on Efficiency

Grafana Loki takes a different philosophy: do not index the full content of logs, only the labels. In practice, it is closer to “Prometheus for logs” than to Elasticsearch: queries by labels and subsequent filtering over lines with LogQL.

Its storage is based on compressed chunks and a small index. Chunks usually live in object storage such as S3, GCS, Azure Blob or MinIO, while the index points to streams defined by label combinations. That decision reduces cost, but shifts responsibility to label design.

The practical rule is simple: labels must have controlled cardinality. cluster, namespace, app, environment or service usually work well. request_id, user_id or trace_id as labels can break the model because they multiply streams, hurt compaction and grow the index. For those values, it is better to filter log content or rely on traces.

Loki shines in Kubernetes because the label model fits pods, namespaces and services. Its Grafana integration is also excellent: log exploration, jumps from metrics, alerts with LogQL and correlation with Tempo or Prometheus.

Its weakness appears when you want to treat logs as an arbitrary search corpus. A query like “find this fragment across all logs from the last six months” can be much more expensive than in a system with a full-text index. Loki is designed for queries bounded by labels and time, not to replace a general-purpose search engine.

In 2026, Loki remains highly active. The official documentation lists recent 3.x versions, including the 3.7 branch, and Grafana continues to position it as a horizontally scalable, highly available and cost-efficient log aggregation system. Grafana’s acquisition of Logline further reinforces the focus on difficult searches within logs, although it does not change the central principle: label design remains critical.

When Loki Shines

Loki is the pragmatic choice for Kubernetes teams that already live in Grafana, need to retain volume at low cost and query by service, namespace, cluster, environment and time window. It is less convincing for global full-text search or deep forensic analysis.

Quickwit: Cloud-Native Search on Object Storage

Quickwit is a distributed search platform written in Rust, designed for append-only workloads such as logs and traces. Its thesis is attractive: keep powerful search, but decouple compute and storage by using object storage as the main layer.

Instead of depending on hot local disks, Quickwit writes indices and splits to S3-compatible storage, Azure Blob, Google Cloud Storage or MinIO. Indexers process data, searchers query segments and the metastore coordinates state. The model uses inverted indices optimized for immutable data and remote storage: closer to OpenSearch in search than Loki, but with cloud-native economics.

Quickwit is especially interesting for teams that want to replace part of an expensive Elasticsearch/OpenSearch cluster, already have logs in S3 or can move them there, and need text search with a better operational cost profile. It also has Grafana integration and APIs that are partially compatible with Elasticsearch/OpenSearch, which helps with some existing workflows.

The critical nuance for 2026 is its project status. Datadog acquired Quickwit in 2024. The acquisition did not mean the open source repository disappeared: the quickwit-oss/quickwit repository remains public and shows recent activity, including 2026 tags and discussions. The Quickwit Grafana plugin also publishes recent compatibility with Grafana 12.1+ and 13. That said, the product risk is different from Loki or OpenSearch: strategic direction is influenced by Datadog, and part of the effort may be oriented toward internal or commercial use cases.

The honest reading is this: Quickwit should not be dismissed as “dead,” because according to the reviewed sources it is not archived. But it should be evaluated with more diligence: release cadence, critical issues, roadmap, real API compatibility, operational ease and dependency on the maintainer team. For a core observability platform, that evaluation matters as much as the benchmark.

ClickHouse: The Analytical Engine of New Observability

ClickHouse was born as a columnar analytical database, not as a log platform. That is exactly why it has become relevant to observability. Many incidents are not solved by searching for an exact string, but by grouping by service, calculating percentiles, exploring cardinalities, joining logs with traces and querying billions of events over a time range.

ClickHouse’s model is columnar. Data is stored by columns, compresses extremely well and is queried with vectorized execution, ordered primary indices, data skipping indexes and engines such as MergeTree. For observability, this makes it possible to store wide events and read only the columns needed.

The tradeoff is that ClickHouse is not Elasticsearch. Full-text search exists and has improved, but its strength remains structured and semi-structured analysis. If your queries are “give me all 500 errors by version, region and endpoint in the last 15 minutes,” ClickHouse is a very good fit. If your main query is “find any line containing a rare phrase across six months of unstructured text,” you probably want another component or a hybrid strategy.

ClickHouse has moved decisively toward observability. The acquisition of HyperDX and the launch of ClickStack change its positioning: it is no longer only “use ClickHouse as the backend and build the UI yourself,” but a stack with OpenTelemetry, HyperDX as the interface and ClickHouse as the engine. It shines in high-volume platforms, structured logs, traces, derived metrics and low-latency aggregate queries, although it requires understanding modeling, TTLs, compression and ingestion costs.

Quick Comparison

CharacteristicOpenSearchLokiQuickwitClickHouse
Main modelDistributed search on LuceneLogs by streams and labelsCloud-native distributed searchColumnar analytical database
Storage modelShards, Lucene segments, hot/warm/cold tiersCompressed chunks + small indexIndices/splits in object storageCompressed columns in MergeTree
IndexInverted, very completeLabels, not full contentInverted, optimized for object storagePrimary index, data skipping, columns
Full-text searchExcellentLimited and dependent on filtersVery goodGood, not its main strength
Relative costHighLowLow-mediumLow-medium
Query latencyLow on well-sized hot indicesLow if bounded by labels; worse on broad scansLow-medium depending on cache and object storageVery low on well-modeled aggregations
ScalingMature, but operationally involvedHorizontal and economicalDecouples compute and storageExcellent for analytics and ingestion
KubernetesGoodExcellentGoodGood, better with ClickStack/OTel
GrafanaYesNativeYesYes
Elasticsearch compatibilityHigh in OpenSearchNoPartialNo
Best use caseComplex search and compatibilityLow-cost Kubernetes logsEfficient search on S3Unified analytical observability
Main riskCost and complexityLabel cardinalityEcosystem and governance after acquisitionModeling and SQL/operations learning curve

Which Should You Choose in 2026?

The answer depends less on technology fashion and more on the team’s profile.

If You Are a Platform Team with Elasticsearch Heritage

OpenSearch is the conservative option if you already have automation, alerts and operational knowledge around Elasticsearch/OpenSearch. Do not migrate “as is” expecting costs to drop: review mappings, retention, ILM/ISM, shards, indexed fields and the separation between hot data and archive.

If You Are a Kubernetes Team Centered on Grafana

Loki is usually the first option. It is efficient, integrates very well with Grafana and reduces the cost of operational logs. The key is to govern labels from day one: allowed taxonomy, cardinality limits and context-driven queries before text searches.

If You Want to Reduce Cost Versus OpenSearch Without Losing Search

Quickwit deserves a serious PoC when Loki falls short on search and OpenSearch is expensive. The decision must include continuity: repository activity, releases, clients, community support, Grafana integration and Datadog’s position.

If You Want a Unified Observability Platform

ClickHouse is probably the strongest bet for consolidating logs, traces, derived metrics and events in a shared analytical database. ClickStack and HyperDX reduce the need to build the entire experience from scratch, but adoption depends on maturity in SQL, modeling and analytical database operations.

If You Have Security, Audit or SIEM Requirements

OpenSearch remains more natural for textual investigation and security tooling. ClickHouse can work for security analytics at scale, but it requires modeling. Loki is rarely the first choice for SIEM. Quickwit can be interesting, with a smaller ecosystem.

If Budget Is the Bottleneck

Do not choose only by storage cost per TB. Calculate total cost: ingestion, queries, retention, operations, backups, upgrades, training and incident time. Loki and ClickHouse usually win on raw cost; Quickwit can be highly competitive on object storage; OpenSearch can be reasonable if it is limited to data that is truly searchable and hot.

How to Frame a Decision PoC

A useful test is not installing four Helm charts and looking at a demo. Define real questions from past incidents and run them against representative data:

  • The last 7 days of production logs, with real cardinality.
  • A hot 2-hour window with high volume.
  • A rare text search over a broad period.
  • An aggregation by service, endpoint, version and region.
  • A correlation flow from alert to logs and traces.

Measure sustained ingestion, storage cost, p50/p95 latency, operational complexity and team experience. Include an uncomfortable scenario: a failed node, slow object storage, unexpected cardinality or a badly written query.

Conclusion

The era when Elasticsearch was the only viable option for logs is over. In 2026, there are four clear paths.

OpenSearch is the mature choice for full-text search and compatibility. Loki is efficient for operational logs in Kubernetes. Quickwit offers modern search on object storage, with mandatory governance evaluation after the Datadog acquisition. ClickHouse is consolidating as an analytical engine for unified observability, especially with ClickStack and HyperDX.

The best platform is not the one that wins every benchmark. It is the one that answers your production questions with the lowest sustainable cost and the least operational friction for your team.

CTA: Decide with Data, Not Preferences

Before migrating or standardizing, run a two-week PoC with real data, real queries and an honest estimate of total cost. It should reveal which data belongs in search, which belongs in analytics and which belongs in cheap storage.

Sources

  • OpenSearch 3.0: https://opensearch.org/blog/unveiling-opensearch-3-0/
  • OpenSearch 3.0, Lucene 10 and JVM 21: https://opensearch.org/blog/opensearch-3-0-what-to-expect/
  • Grafana Loki, documentation: https://grafana.com/docs/loki/latest/
  • Grafana Loki, releases 3.x: https://grafana.com/docs/loki/latest/release-notes/
  • Grafana Loki, labels and cardinality: https://grafana.com/blog/how-labels-in-loki-can-make-log-queries-faster-and-easier/
  • Datadog acquires Quickwit: https://www.datadoghq.com/blog/datadog-acquires-quickwit/
  • Quickwit OSS: https://github.com/quickwit-oss/quickwit
  • Quickwit datasource for Grafana: https://github.com/quickwit-oss/quickwit-datasource
  • ClickStack: https://clickhouse.com/clickstack
  • ClickStack/HyperDX in ClickHouse Cloud: https://clickhouse.com/docs/use-cases/observability/clickstack/overview
  • ClickHouse acquires HyperDX: https://clickhouse.com/blog/202504-newsletter
  • ClickHouse and observability 2026: https://clickhouse.com/resources/engineering/what-is-observability

Kamal vs Kubernetes: An Honest Comparison for Teams Who Don’t Need 1,000 Services

Kamal vs Kubernetes: An Honest Comparison for Teams Who Don't Need 1,000 Services

The framing problem

Most “Kamal vs Kubernetes” articles are written by people who just discovered Kamal and are excited about it. They frame it as a simpler alternative that embarrasses Kubernetes. That framing is wrong and it will lead you to the wrong decision.

Kamal is not better than Kubernetes. Kubernetes is not overkill for everyone. They solve different problems at different scales, and picking the wrong one for your context is expensive either way.

This article gives you the honest version: what each tool actually is, where each one breaks, and the specific signals that should push you in one direction or the other.


What Kamal actually is

Kamal (formerly MRSK) is a deployment tool built by 37signals. It deploys Docker containers to servers via SSH, uses kamal-proxy as its default reverse proxy, and handles rolling deploys with zero downtime. That is the entire feature set.

kamal deploy

It connects to your servers over SSH, pulls the new image, starts the new container, waits for it to be healthy, then moves traffic through kamal-proxy and stops the old one. Traefik is still possible, but in Kamal 2 it sits in front of kamal-proxy as an optional accessory rather than being the standard path.

There is no control plane. No etcd. No API server. No scheduler. No concept of desired state reconciliation. Docker or systemd can restart a crashed container if you configure the right restart policy, but Kamal itself does not reschedule workloads to another node or provide cluster-level self-healing.

Kamal is a deployment tool. Kubernetes is an orchestration platform. These are not the same category.

That is the key distinction from tools like Nomad. I covered that angle in Nomad vs Kubernetes, as part of the same “lighter alternatives to Kubernetes” discussion. Nomad is a scheduler/orchestrator. Kamal is a deployment tool that leaves scheduling and node-level recovery mostly outside its scope.


What Kubernetes actually is

Kubernetes is a distributed system that continuously reconciles actual state against desired state. You declare what you want — N replicas of container X, with these resource limits, this health check, this rollout strategy — and the control plane makes it happen and keeps it that way.

When a node fails, your pods are rescheduled. When a container crashes, it is restarted. When you push a new image, the rollout respects your maxUnavailable and maxSurge settings. When traffic spikes, HPA can add replicas.

This machinery is powerful. It is also genuinely complex to operate.

A minimal production-grade Kubernetes cluster involves: a multi-node control plane for HA, a CNI plugin, an ingress controller (or Gateway API), cert-manager for TLS, a CSI driver for storage, a metrics server for HPA, RBAC policies, network policies, pod disruption budgets, and some form of secret management. Getting all of this right takes time and ongoing maintenance.

The question is whether the problems Kubernetes solves are problems you actually have.


The minimum config tells the story

Here is the same tiny web service expressed in both worlds. The exact fields vary by app, registry, ingress setup, and Kubernetes distribution, but the shape is representative.

<table> <thead> <tr> <th>Kamal <code>deploy.yml</code></th> <th>Kubernetes <code>Deployment</code> + <code>Service</code></th> </tr> </thead> <tbody> <tr> <td> <pre><code class=”language-yaml”>service: web image: ghcr.io/acme/web

servers: web: hosts: – 203.0.113.10

proxy: ssl: true host: app.example.com app_port: 3000

registry: username: acme password: – KAMAL_REGISTRY_PASSWORD

env: secret: – DATABASE_URL</code></pre> </td> <td> <pre><code class=”language-yaml”>apiVersion: apps/v1 kind: Deployment metadata: name: web spec: replicas: 2 selector: matchLabels: app: web template: metadata: labels: app: web spec: containers: – name: web image: ghcr.io/acme/web ports: – containerPort: 3000 envFrom: – secretRef: name: web-secrets — apiVersion: v1 kind: Service metadata: name: web spec: selector: app: web ports: – port: 80 targetPort: 3000</code></pre> </td> </tr> </tbody> </table>

The Kubernetes example is still incomplete for internet traffic and TLS: you would normally add an Ingress or HTTPRoute, cert-manager, DNS, and secret provisioning. Kamal’s example is closer to “deploy this app to these servers.” Kubernetes is closer to “declare this workload inside a platform that already exists.”


The 37signals case

37signals (the company behind Basecamp and Hey) moved off cloud Kubernetes to bare-metal servers running Kamal in 2022-2023. Their own Ops team wrote about the “de-cloud and de-k8s” move. The story got a lot of attention.

What the hot takes missed: 37signals is not a startup. They have a dedicated ops team. They own and operate hardware. Their workload is well-understood and stable: a small number of mature applications serving a large, established user base.

Their decision was rational for their context. Kubernetes was costing them in cloud spend and operational complexity for problems their workload does not have. Moving to Kamal on owned hardware made economic and operational sense for them.

It does not follow that Kamal is the right choice for your early-stage startup, your company that runs on AWS because your team does not want to manage hardware, or your platform that needs to scale from 0 to 10x on short notice.


Where Kamal breaks

No cluster-level self-healing. If your container crashes, Docker or systemd can restart it with the right policy, but Kamal has no involvement. If your server fails, Kamal will not reschedule that workload elsewhere; your service is down until you intervene or until you have set up something else to handle it.

No built-in auto-scaling. Kamal has no equivalent of HPA or KEDA. If your traffic spikes, you add servers manually or script something yourself.

No multi-AZ HA by default. You can deploy to multiple servers across regions, but the coordination is yours to manage. There is no concept of spreading replicas across availability zones automatically.

Rolling deploys are basic. Kamal’s zero-downtime deploy is good for single-server or simple multi-server setups. It is not a substitute for Kubernetes rolling update with fine-grained health check integration across a fleet.

Secret management is manual. Kamal reads secrets from .kamal/secrets and has kamal secrets helpers for fetching and extracting values from password managers and cloud secret stores, but there is no Kubernetes-style projected secret lifecycle. Rotation and policy remain yours to manage.

No workload isolation. Everything on the same server shares the same kernel, same resources. Resource limits are Docker-level, not enforced by an orchestration layer with quotas and namespaces.


Where Kubernetes breaks

The complexity tax is real. Running Kubernetes yourself — not a managed service, but actual cluster operation — requires dedicated expertise. A team without a platform engineer will spend significant time debugging networking issues, understanding why pods are Pending, or figuring out why cert-manager is not issuing certificates.

Managed K8s is not free. EKS charges for the cluster control plane, with standard support priced around $0.10/hour per cluster; GKE and AKS have their own pricing rules and credits. In all cases, you still pay for nodes, load balancers, and persistent volumes. For a small application, this cost can exceed what bare-metal hosting would cost.

The YAML surface area scales with complexity. A simple web app on Kubernetes requires a Deployment, a Service, an Ingress (or HTTPRoute), a Certificate, potentially a HorizontalPodAutoscaler, a PodDisruptionBudget, and NetworkPolicies. That is a lot of infrastructure for a CRUD app.

Debugging is harder. Distributed systems are harder to debug than single-server setups. kubectl exec and kubectl logs are good tools but diagnosing why traffic is not reaching your pod — DNS, CNI, network policy, service selector, readiness probe — requires Kubernetes-specific knowledge that takes time to build.


The honest decision framework

Ask yourself these questions in order:

1. Do you have more than one team working on independent services with different scaling needs?

If yes: Kubernetes starts making sense. The isolation and per-service resource management are worth the overhead.

If no: Kamal is probably sufficient.

2. Do you need to survive individual server failures automatically, without manual intervention?

If yes: You need either Kubernetes with proper HA setup, or a managed container platform whose recovery and placement model fits your app. Cloud Run, Fly.io, Render, and Railway all help here, but their HA boundaries differ. Kamal alone does not give you this.

If no: A Docker or systemd restart policy covers the common crash case.

3. Does your traffic profile require automatic scaling?

If yes: Kubernetes with HPA or KEDA, or a managed platform with the scaling behavior you actually need. Cloud Run scales request-driven services to and from zero; Fly.io supports autostart/autostop and machine scaling; Render and Railway support scaling, but with provider-specific limits and billing. Kamal has no answer here without custom tooling.

If no: Fixed capacity is fine. Kamal works.

4. Do you have a platform team, or is “ops” a shared responsibility among product engineers?

If platform team exists: Kubernetes operational overhead is distributed and manageable.

If ops is everyone’s side job: Kamal’s simplicity is a genuine advantage. Less to go wrong, less to learn, faster to debug.

5. Are you on cloud infrastructure you do not own, and cloud costs matter?

If yes: Run the numbers on managed K8s vs a few VPS instances. For small workloads, VPS + Kamal is significantly cheaper.

If no: You have hardware already. Kamal is almost certainly the right choice.


Side-by-side

KamalManaged container platformsKubernetes
ExamplesKamal on VPS/bare metalCloud Run, Fly.io, Render, RailwayEKS, GKE, AKS, self-managed
Deployment modelSSH + DockerPlatform API / Git deploy / container deployAPI-driven, reconciliation loop
Self-healingDocker/systemd restart policy onlyPlatform-managed restart and placementPod rescheduling, node failure recovery
Auto-scalingManualProvider-specific; not all scale the same wayHPA, VPA, KEDA
HAManual (multiple servers)Provider-managed, with platform limitsAvailable with a well-configured multi-node control plane (not free)
TLSkamal-proxy with automatic HTTPS; Traefik optionalUsually built incert-manager + ingress controller
Secret management.kamal/secrets + helper commandsPlatform secretsSecrets API, external-secrets, Vault
ObservabilityStandard Docker loggingProvider logs/metrics, export variesRich ecosystem (Prometheus, OTel, etc.)
Rollout controlBasic rolling deploySimple rollouts, provider-specificFine-grained (maxUnavailable, canary, etc.)
Learning curveLowLow to mediumHigh
Ops overheadLowLowMedium to high
Right scale1-20 services, stable loadSmall to medium teams that want managed recovery/scaling10+ services, variable load, or regulated

What the threshold actually looks like

Kamal is the right default if you are:

  • A team of 2–10 engineers shipping a web application
  • Running stable, predictable workloads
  • On a budget where managed K8s costs matter
  • Without a dedicated platform engineer

Kubernetes is the right choice when:

  • You have 10+ services with independent deployment cycles
  • You need multi-zone HA and automatic failover
  • Traffic is variable enough that auto-scaling saves real money or prevents real incidents
  • You have a platform team that can absorb the operational complexity
  • You are in a regulated environment that benefits from K8s’s audit trails and RBAC

There is a middle tier worth mentioning: managed container platforms like Fly.io, Railway, Render, and Google Cloud Run. These give you some of Kubernetes’s operational benefits with closer to Kamal’s operational simplicity, but the details matter: HA, auto-scaling, scale-to-zero, regions, and billing differ by provider. For teams that need more than Kamal but do not want to operate K8s, this tier is often the right answer and gets underrepresented in the debate.


The real lesson from 37signals

The story is not “Kamal beat Kubernetes.” The story is: match your infrastructure to your actual operational profile, not to what is fashionable or what scales to Google’s size.

37signals evaluated what problems they actually had and picked the tool that solved those problems at the lowest operational cost. That is the right framework.

For most teams reading this, Kubernetes is probably the right answer eventually — when your system’s complexity justifies it. The mistake is adopting it before you are there, burning engineering time on infrastructure problems instead of product problems.

Kamal is a good tool for a specific stage and scale. Use it until you outgrow it. When you outgrow it, you will know — because the things Kubernetes solves will be actual problems you have, not hypothetical ones.


What to do next

Before you choose, write down your actual requirements in one page: number of services, expected traffic variance, failure tolerance, who owns ops, whether you own hardware, and what compliance needs are real today. Then run a small proof of concept with the simplest tool that satisfies those requirements. If Kamal plus Docker restart policies covers the failure modes you actually accept, start there. If you need automatic placement, autoscaling, and multi-zone recovery now, compare managed container platforms first, then Kubernetes.


FAQ

Can Kamal and Kubernetes coexist?

Yes. Some teams use Kamal for simple auxiliary services (internal tools, cron jobs, simple APIs) while running their core platform on Kubernetes. The tools are not mutually exclusive.

Is Kamal production-ready?

Yes. 37signals runs Basecamp and Hey on it. It handles zero-downtime deploys, TLS, and multi-server deployments reliably. The limitation is not reliability — it is feature scope.

What about Docker Swarm? Is it still relevant?

Docker Swarm fills a similar niche — simpler than Kubernetes, multi-host orchestration. Swarm mode is still documented and supported as part of Docker Engine for teams that want it as a production runtime, but its ecosystem is much smaller than Kubernetes and less active in the market. I would not pick it as the default for a new project unless the team already knows and wants Swarm.

Does Kamal work with any cloud provider?

Yes. Kamal only requires SSH access to a server and a container registry. It works with any VPS provider (Hetzner, DigitalOcean, OVH, AWS EC2, etc.) and any registry (Docker Hub, GitHub Container Registry, ECR, Harbor).

Is Kubernetes worth learning even if you use Kamal today?

Yes. Kubernetes is the dominant orchestration platform. Understanding it — even without operating it — makes you a better platform engineer and opens more career options. Learning Kamal does not preclude learning Kubernetes.

Stop Implementing Authentication Inside Containers on Kubernetes

Stop Implementing Authentication Inside Containers on Kubernetes

You have ten microservices running in Kubernetes. Each one validates JWTs, checks scopes, maintains sessions, and implements its own RBAC rules. One team uses jsonwebtoken v8, another uses a custom Go library, a third rolled their own HMAC check because “it was simple.” They all accept alg: none. Three accept RS256 and HS256 simultaneously.

This is not a security posture. This is a distributed security liability — and Kubernetes makes the problem worse, because the cluster creates an illusion of isolation that encourages teams to treat each Pod as a security boundary it was never designed to be.

The pattern of embedding authentication and authorization logic inside individual containers is one of the most pervasive anti-patterns in Kubernetes-based microservices. It feels like ownership and simplicity. It is, in practice, inconsistency at scale — and the blast radius of a single misconfiguration is your entire service portfolio.

Kubernetes provides the primitives to fix this at the infrastructure layer: Ingress controllers, the Gateway API, service mesh sidecars, admission webhooks, and workload identity via SPIFFE. None of these require a line of auth code inside your application containers.

This article explains why the anti-pattern exists, what’s wrong with it technically, and what the correct Kubernetes-native alternatives are — with concrete implementation guidance and references to the standards and incidents that validate the argument.


The Anti-Pattern: What It Looks Like

In-Application JWT Validation

Every service imports an auth library and validates tokens independently:

# Pattern seen in thousands of microservices
from jose import jwt

def authenticate(token: str):
    payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
    return payload["sub"]

Variations include:

  • Algorithm confusion: accepting both HS256 and RS256, or letting the token header drive verification behavior instead of pinning acceptable algorithms server-side. This is a distinct JWT implementation failure class, documented extensively by PortSwigger on JWT attacks and RFC 8725
  • alg: none bypass: libraries that accept unsigned tokens when alg is set to none. This is a documented attack vector in Auth0’s JWT security analysis
  • Missing exp / iss / aud validation: trusting a valid signature without checking whether the token is expired, for the right audience, or from the right issuer
  • Key confusion attacks: accepting a public RS256 key as an HS256 symmetric secret

Session State in Every Service

When services maintain session state directly, they duplicate logic that has no business being duplicated — cookie validation, refresh token flows, PKCE verification — and each implementation diverges over time.

RBAC Reimplemented Per Service

Authorization rules (“can this user access this resource?”) end up embedded in service logic, mixed with business logic, tested inconsistently, and impossible to audit across the portfolio.


Why This Is a Structural Problem

1. The Vulnerability Surface Scales With Your Service Count

Each new microservice is a new JWT validation surface. A single incorrect library configuration — an unvalidated alg, a missing aud check — is an authentication bypass affecting that entire service. With ten services, you have ten potential misconfigurations. With a hundred services, the probability that at least one is misconfigured approaches certainty.

The OWASP Kubernetes Security Cheat Sheet and OWASP Microservices Security Cheat Sheet both identify in-service auth as a primary attack surface in microservices environments. NIST SP 800-204 and its companion NIST SP 800-204A on DevSecOps make the same argument: security controls belong at infrastructure boundaries, not inside application code.

2. Maintenance Cost Is Multiplicative

When a JWT vulnerability is disclosed — and they are disclosed regularly — you update one library in one service. Then another. Then you discover service C is pinned to an old version because it has a transitive dependency conflict. Meanwhile the vulnerability is exploitable in production.

The CNCF Cloud Native Security Whitepaper frames this directly: security controls implemented redundantly across services create maintenance overhead that teams cannot sustain, leading to version drift and policy divergence.

3. Centralized Policy Is Impossible to Enforce

When policy is in code — even well-factored library code — it cannot be changed atomically across services. A policy update requires a coordinated deployment across every affected service. In practice, services deploy on different schedules, managed by different teams, with different testing cycles. The result is that at any given moment, some fraction of your services are running different authorization rules.

This is the core argument in Google’s BeyondCorp model and the Zero Trust Architecture guidance from NIST SP 800-207: authentication and authorization decisions should be made by a centralized, auditable policy enforcement point — not distributed across workloads.

4. Secrets Distribution Is a Problem You Don’t Need

If every service validates JWTs, every service needs the signing key (for symmetric algorithms) or the public key (for asymmetric). Distributing and rotating signing keys across a fleet of microservices is an operational burden with meaningful blast radius: a leaked symmetric key compromises every service holding it.

The CNCF SPIFFE/SPIRE project was built specifically to solve this class of problem: workload identity should be cryptographically attested, not rely on secrets distributed to application code.


The Real-World Incidents

The alg: none Class

In 2015, critical vulnerabilities in JWT libraries from Auth0 affecting Python, PHP, Node.js, Ruby, Java, and .NET allowed attackers to forge tokens by setting alg: none. The signature was not verified. The vulnerability was present in applications that had copied JWT validation code from tutorials or used unpatched libraries — exactly the pattern that in-service auth produces at scale.

Java’s Psychic Signatures (CVE-2022-21449)

CVE-2022-21449 affected ECDSA signature verification in Oracle Java SE and GraalVM, including java.security.Signature paths used by higher-level libraries. The bug allowed certain malformed ECDSA signatures to verify when they should not. JWT validation was in scope only when the deployment used an affected Java runtime and ECDSA-signed tokens, for example ES256. A gateway would help only if verification happened on a patched or unaffected runtime at the gateway instead of inside every Java service.

CVE-2023-2728 (Kubernetes Mountable Secrets Bypass)

CVE-2023-2728 was not an ImagePolicyWebhook outage behavior. It was a Kubernetes API server issue where users could use ephemeral containers to bypass the mountable secrets policy enforced by the ServiceAccount admission plugin. Clusters were affected only when the ServiceAccount admission plugin, the kubernetes.io/enforce-mountable-secrets annotation, and ephemeral containers were used together. The adjacent ImagePolicyWebhook issue was CVE-2023-2727, also involving ephemeral containers, but it is a separate CVE.

The Uber API Gateway Evolution

Uber’s engineering blog describes its API gateway as a centralized layer for routing, protocol conversion, rate limiting, load shedding, header propagation, security auditing, and user access blocking. That supports the architectural point here: high-volume platforms move cross-cutting controls into shared infrastructure. The public source does not prove that Uber migrated specifically from per-service authentication to gateway authentication, so that stronger claim should not be made.

Netflix Zuul

Netflix’s Zuul is an L7 gateway for dynamic routing, monitoring, resiliency, security, and related edge concerns. Netflix’s own Zuul posts list authentication among common edge-service uses, but they do not frame Zuul primarily as a case study in eliminating per-service auth. Treat it as evidence that authentication is a natural edge concern at scale, not as proof of a specific migration story.


The Alternatives

Use these as complementary controls, not as a single replacement for all authentication and authorization logic:

AlternativeWhen to use itWhat it solvesPrincipal trade-off
API Gateway / Edge AuthExternal API clients, public ingress, partner integrations, mixed auth mechanisms at the boundaryCentral JWT/API-key/OAuth2 validation, rate limiting, request shaping, and identity header propagation before traffic reaches servicesDoes not secure east-west service calls by itself; trusted headers require strict network boundaries
Service Mesh mTLSService-to-service traffic inside the cluster, especially across teams or sensitive domainsWorkload identity, automatic mTLS, peer authentication, and proxy-level authorization policyAdds data-plane/control-plane complexity and operational coupling to sidecars or ambient mesh components
OAuth2 ProxyBrowser-facing internal apps that need OIDC login, redirects, cookies, and session handlingDelegates login/session management to a reverse proxy and forwards authenticated identity headersBest for HTTP/browser flows; not a general machine-to-machine authorization system
OPAComplex, auditable, frequently changing authorization rulesSeparates policy decisions from application releases and can run as sidecar, service, ext_authz backend, or admission control via GatekeeperPolicy/data distribution and failure behavior must be designed deliberately
SPIFFE/SPIREMulti-cluster, multi-cloud, or meshless environments that need portable workload identityIssues short-lived workload identities without application-managed shared secretsProvides identity, not business authorization; needs registration and attestation lifecycle management

Option 1: API Gateway (Edge Auth)

An API Gateway sits at the perimeter and handles authentication before a request reaches any downstream service. Services receive pre-validated identity in a trusted header.

What it does: – Validates JWTs, API keys, OAuth2 tokens – Enforces rate limiting per identity – Strips and re-adds Authorization headers as needed – Routes to upstream services with verified identity headers

When to use it: – North-south traffic (external clients → cluster) – Mixed authentication mechanisms (JWT + API key + mTLS) at the Ingress layer – Teams that want to centralize auth policy without rolling out a full service mesh

Tools:

Gravitee.io API Gateway can be deployed on Kubernetes via its Helm chart and integrates with the Kubernetes Gateway API:

helm repo add graviteeio https://helm.gravitee.io
helm install gravitee-apim graviteeio/apim \
  --namespace gravitee \
  --create-namespace \
  --set gateway.replicaCount=2 \
  --set gateway.ingress.enabled=true \
  --set gateway.ingress.hosts[0]=api.example.com

Once deployed, authentication policies are declared on the ApiV4 CRD — no application code involved:

apiVersion: gravitee.io/v1alpha1
kind: ApiV4
metadata:
  name: payment-api
  namespace: gravitee
spec:
  name: "Payment API"
  type: PROXY
  listeners:
    - type: HTTP
      paths:
        - path: /v1/payments
      entrypoints:
        - type: http-proxy
  endpointGroups:
    - name: default
      type: http-proxy
      endpoints:
        - name: upstream
          type: http-proxy
          configuration:
            target: http://payment-service.production.svc.cluster.local:8080
  flows:
    - name: JWT validation
      enabled: true
      request:
        - policy: jwt
          enabled: true
          configuration:
            signature: RSA_RS256
            publicKeyResolver: JWKS_URL
            jwksUrl: https://idp.example.com/.well-known/jwks.json
            checkTokenRevocation: true
            requiredClaims:
              - name: aud
                value: payment-api
        - policy: rate-limit
          enabled: true
          configuration:
            rate:
              limit: 100
              periodTime: 1
              periodTimeUnit: MINUTES

Gravitee’s Kubernetes operator reconciles ApiV4 resources against the gateway, making API policy a first-class GitOps object — versioned, reviewed, and deployed the same way as any other Kubernetes manifest.

Other Kubernetes-native options: Emissary-Ingress (formerly Ambassador) with its AuthService CRD; Traefik with ForwardAuth middleware on IngressRoute resources.

Limitations: API Gateways handle north-south traffic. They don’t address east-west (service-to-service) authentication inside the cluster.


Option 2: Service Mesh (East-West mTLS + Auth)

A service mesh provides mutual TLS between every service pair and enforces authorization policy at the sidecar proxy, without any application code changes.

What it does: – Automatic mTLS between all service-to-service calls – Workload identity via X.509 certificates (SPIFFE SVIDs) – Fine-grained AuthorizationPolicy at the Envoy sidecar – JWT validation at the proxy, not the application

Istio Implementation:

Istio uses Envoy’s ext_authz filter and native RequestAuthentication + AuthorizationPolicy CRDs:

# RequestAuthentication — validate JWTs at the proxy
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: require-jwt
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-service
  jwtRules:
    - issuer: "https://accounts.google.com"
      jwksUri: "https://www.googleapis.com/oauth2/v3/certs"
      audiences:
        - "my-api-audience"
      forwardOriginalToken: false
---
# AuthorizationPolicy — enforce after JWT validation
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payment-service-authz
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/production/sa/order-service"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/payments"]
      when:
        - key: request.auth.claims[scope]
          values: ["payments:write"]

The RequestAuthentication policy tells Envoy how to validate JWTs. The AuthorizationPolicy specifies what authenticated principals are allowed to do. Neither policy lives in application code.

The payment service receives the validated request — or a 401/403 from the proxy, before the request touches application code.

Linkerd:

Linkerd provides automatic mTLS with SPIFFE-compliant workload identity. Its policy model is simpler than Istio but sufficient for most service-to-service auth requirements:

apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata:
  name: payment-server
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: payment-service
  port: 8080
---
apiVersion: policy.linkerd.io/v1beta3
kind: ServerAuthorization
metadata:
  name: order-to-payment
  namespace: production
spec:
  server:
    name: payment-server
  client:
    meshTLS:
      serviceAccounts:
        - name: order-service

This is mutual TLS + SPIFFE-based identity, enforced at the proxy. The application doesn’t implement it; the mesh does.

Istio + Envoy External Authorization:

For more complex policy (e.g., OPA integration), Envoy’s ext_authz filter delegates authorization to an external service:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz-filter
  namespace: production
spec:
  workloadSelector:
    labels:
      app: payment-service
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ext_authz
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
            grpc_service:
              envoy_grpc:
                cluster_name: outbound|9191||opa.production.svc.cluster.local
            timeout: 0.5s
            failure_mode_allow: false

The CNCF TAG Security paper on microservices security documents this architecture as the reference pattern for production Kubernetes environments.


Option 3: OAuth2 Proxy (Delegated Auth for HTTP)

OAuth2 Proxy is a reverse proxy that authenticates requests against an OAuth2/OIDC provider and passes validated identity downstream. With 14,000+ GitHub stars and active maintenance, it is the most widely deployed solution for this pattern in Kubernetes.

What it does: – Sits in front of one or more upstream services – Redirects unauthenticated requests to an OIDC provider (Keycloak, Dex, Google, GitHub, etc.) – Validates tokens, manages sessions, handles refresh – Passes X-Auth-Request-User, X-Auth-Request-Email, X-Auth-Request-Groups headers downstream

Kubernetes deployment with Nginx Ingress:

# OAuth2 Proxy deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oauth2-proxy
  namespace: auth
spec:
  replicas: 2
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
        - name: oauth2-proxy
          image: quay.io/oauth2-proxy/oauth2-proxy:v7.6.0
          args:
            - --provider=oidc
            - --oidc-issuer-url=https://keycloak.example.com/realms/myrealm
            - --client-id=$(CLIENT_ID)
            - --client-secret=$(CLIENT_SECRET)
            - --cookie-secret=$(COOKIE_SECRET)
            - --http-address=0.0.0.0:4180
            - --reverse-proxy=true
            - --upstream=static://202
            - --email-domain=*
            - --set-xauthrequest=true
            - --cookie-secure=true
            - --skip-provider-button=true
          env:
            - name: CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: oauth2-proxy-secrets
                  key: client-id
            - name: CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: oauth2-proxy-secrets
                  key: client-secret
            - name: COOKIE_SECRET
              valueFrom:
                secretKeyRef:
                  name: oauth2-proxy-secrets
                  key: cookie-secret
---
# Ingress annotation to protect a service with OAuth2 Proxy
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: protected-service
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "https://oauth2-proxy.example.com/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://oauth2-proxy.example.com/oauth2/start?rd=$escaped_request_uri"
    nginx.ingress.kubernetes.io/auth-response-headers: "X-Auth-Request-User,X-Auth-Request-Email,X-Auth-Request-Groups"
spec:
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: protected-service
                port:
                  number: 8080

The upstream service receives X-Auth-Request-User and X-Auth-Request-Groups as trusted headers — it never sees a token, never validates a signature, never imports a JWT library.

When to use OAuth2 Proxy vs a service mesh: OAuth2 Proxy handles north-south browser-facing traffic with session management (login flows, redirects, cookies). A service mesh handles east-west machine-to-machine auth. They are complementary, not alternatives.


Option 4: Open Policy Agent (OPA / OPAL)

OPA decouples policy from code entirely. Authorization logic is written in Rego and evaluated by OPA as a sidecar or as a centralized service. Applications query OPA for allow/deny decisions.

# Rego policy — payment service authorization
package payments.authz

import future.keywords.if
import future.keywords.in

default allow := false

allow if {
    input.method == "POST"
    input.path == "/v1/payments"
    "payments:write" in input.token.scope
    input.token.iss == "https://accounts.example.com"
}

allow if {
    input.method == "GET"
    startswith(input.path, "/v1/payments/")
    "payments:read" in input.token.scope
}

Application code becomes:

// The ONLY auth code in the application
func authMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        input := map[string]interface{}{
            "method": r.Method,
            "path":   r.URL.Path,
            "token":  extractToken(r),
        }
        
        result, err := opaClient.Decision(r.Context(), "payments/authz", input)
        if err != nil || !result.Allow {
            http.Error(w, "Forbidden", http.StatusForbidden)
            return
        }
        next.ServeHTTP(w, r)
    })
}

OPAL (Open Policy Administration Layer) adds real-time policy and data updates to OPA deployments — policy changes propagate to all OPA instances within seconds without redeployment.

Kubernetes deployment patterns for OPA:

As a sidecar — OPA runs in the same Pod as the application, evaluating policy over a local socket. Zero network hop, no external dependency:

# Pod template fragment
spec:
  containers:
    - name: payment-service
      image: payment-service:latest
    - name: opa
      image: openpolicyagent/opa:0.63.0
      args:
        - run
        - --server
        - --addr=localhost:8181
        - /policy
      volumeMounts:
        - name: opa-policy
          mountPath: /policy
          readOnly: true
  volumes:
    - name: opa-policy
      configMap:
        name: payment-policy

As a centralized service with Envoy ext_authz — OPA exposes a gRPC endpoint that Istio’s Envoy sidecar calls for every request. Policy is enforced at the proxy, before the application receives the request. This is the pattern used alongside Istio’s EnvoyFilter shown in the service mesh section above.

As OPA GatekeeperOPA Gatekeeper runs as a Kubernetes admission webhook and enforces policies at deploy time, not at runtime. It’s the right tool for preventing misconfigured workloads from being deployed — for example, rejecting any Pod spec that sets hostNetwork: true or defines auth-related environment variables directly. This is complementary to runtime auth enforcement.

OPA is used at scale by Atlassian, Goldman Sachs, Netflix, Chef, and many others, documented in OPA’s production deployments. The CNCF OPA project graduated in 2021.


Option 5: SPIFFE/SPIRE (Workload Identity)

SPIFFE (Secure Production Identity Framework for Everyone) and SPIRE solve the problem of how workloads prove their identity without distributing secrets.

SPIRE issues short-lived X.509 SVIDs (SPIFFE Verifiable Identity Documents) to workloads. Each SVID encodes a SPIFFE URI:

spiffe://example.org/ns/production/sa/payment-service

Services authenticate each other using mTLS with these certificates. No JWT library. No shared secret. No secret distribution problem. Certificate rotation happens automatically every few hours.

# SPIRE Agent DaemonSet on Kubernetes
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: spire-agent
  namespace: spire
  labels:
    app: spire-agent
spec:
  selector:
    matchLabels:
      app: spire-agent
  template:
    metadata:
      labels:
        app: spire-agent
    spec:
      serviceAccountName: spire-agent
      hostPID: true
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: spire-agent
          image: ghcr.io/spiffe/spire-agent:1.9.0
          args: ["-config", "/run/spire/config/agent.conf"]
          volumeMounts:
            - name: spire-config
              mountPath: /run/spire/config
              readOnly: true
            - name: spire-agent-socket
              mountPath: /run/spire/sockets
              readOnly: false
      volumes:
        - name: spire-config
          configMap:
            name: spire-agent
        - name: spire-agent-socket
          hostPath:
            path: /run/spire/sockets
            type: DirectoryOrCreate

SPIFFE is the foundation of Istio’s workload identity model. Linkerd implements SPIFFE-compatible SVIDs. If you’re using a service mesh, you already have SPIFFE — the mesh uses it transparently.

Standalone SPIRE is appropriate for environments without a service mesh, or for multi-cluster/multi-cloud scenarios where a consistent workload identity layer is needed across boundaries.

SPIFFE/SPIRE graduated from the CNCF sandbox to incubating in 2019 and is deployed at Uber, Bloomberg, ByteDance, and Anthem.


Combining the Layers

These tools are not mutually exclusive — they address different traffic patterns and different problems:

LayerToolAddresses
Edge (north-south)API Gateway (Gravitee, Emissary, AWS APIGW)External clients → cluster
Browser sessionsOAuth2 ProxyBrowser-facing apps with login flows
East-west mTLSService Mesh (Istio/Linkerd) + SPIFFEService-to-service identity
PolicyOPA/OPALFine-grained, auditable authorization
Workload identitySPIREMulti-cloud/multi-cluster identity

A production deployment at reasonable scale looks like:

  1. External traffic hits an API Gateway or Ingress controller with OAuth2 Proxy
  2. The gateway validates the token, strips it, and forwards identity headers to the upstream service
  3. Inside the cluster, all service-to-service calls are mTLS via a service mesh, using SPIFFE workload identity
  4. Authorization decisions (beyond identity) are delegated to OPA
  5. Application code contains zero JWT validation, zero session management, zero auth library imports

Migration Path

If you already have auth embedded in services, migration doesn’t require a big bang rewrite.

Phase 1: Introduce the Gateway

Deploy an API Gateway or OAuth2 Proxy at the edge. Initially, services continue to validate tokens themselves as a backup — the gateway validates first. Use this phase to verify the gateway’s behavior and build confidence.

Phase 2: Trust the Gateway

Add a service-level feature flag: if a trusted X-Auth-Request-User header is present (set by the gateway), skip internal JWT validation. This decouples service auth from gateway rollout.

Phase 3: Remove In-Service Auth

Once all entry points are covered by the gateway and you have confidence in its reliability, remove the auth code from services. This is the step that actually reduces your attack surface.

Phase 4: Add East-West (Optional)

If east-west service-to-service calls exist and carry sensitive data, introduce a service mesh for mTLS. This is a separate effort from gateway auth and can proceed independently.


Decision Framework

External traffic entering the cluster (Ingress / Gateway API)?
├── Browser-facing app with login flow → OAuth2 Proxy (Nginx Ingress annotations)
└── API clients with tokens → API Gateway (Gravitee ApiV4 / Emissary AuthService)

Service-to-service calls inside the cluster (east-west)?
├── mTLS + identity sufficient → Service Mesh (Istio/Linkerd)
├── Identity across multiple clusters → SPIFFE/SPIRE standalone
└── Istio + fine-grained policy → RequestAuthentication + OPA via ext_authz

Complex, auditable authorization logic?
└── OPA as sidecar or ext_authz (runtime) + OPA Gatekeeper (admission)

Preventing misconfigured workloads from being deployed?
└── OPA Gatekeeper admission webhook

What to Keep in Application Code

Not everything should be removed. The correct model is:

  • Remove: JWT signature verification, token parsing, OAuth2 flows, session management
  • Keep: Business-level authorization (“can this user edit this specific resource?”), assuming identity is provided by infrastructure
  • Keep: Authorization errors surfaced correctly (403 vs 401, meaningful error bodies)
  • Keep: Structured logging of authorization decisions for audit trails

The application receives an authenticated identity from infrastructure. What the application does with that identity — which records to show, which operations to allow based on ownership — is correctly application logic.


Audit Checklist: Moving Auth Out of Containers

Use this as a practical exit checklist for the anti-pattern:

  • Inventory every service that imports JWT, OAuth2, OIDC, session, or custom RBAC libraries.
  • Classify each entry point as north-south, browser session, east-west service call, deploy-time admission policy, or business authorization.
  • Put one enforcing control in front of each class: API Gateway or OAuth2 Proxy for ingress, service mesh mTLS for east-west, OPA for shared policy, and SPIFFE/SPIRE for portable workload identity.
  • Pin JWT issuers, audiences, algorithms, and JWKS sources in infrastructure policy; do not let application code infer them from token headers.
  • Strip client-supplied identity headers at the edge and re-add trusted identity headers only after verification.
  • Define failure behavior explicitly: fail closed for authentication and authorization, and document any temporary fail-open exception with an owner and expiry date.
  • Remove in-service token verification only after every ingress path is covered, logs prove the infrastructure control is enforcing, and rollback has been tested.
  • Keep resource-level business authorization in application code, but feed it an identity established by infrastructure.

References

Standards and Frameworks – NIST SP 800-204: Security Strategies for MicroservicesNIST SP 800-204A: Building Secure Microservices-based Applications Using Service-Mesh ArchitectureNIST SP 800-207: Zero Trust ArchitectureOWASP Kubernetes Security Cheat SheetOWASP Microservices Security Cheat SheetCNCF Cloud Native Security Whitepaper v2RFC 7519: JSON Web Token (JWT)RFC 8725: JSON Web Token Best Current Practices

Vulnerabilities and Attack Classes – CVE-2022-21449: Java Psychic Signatures (ECDSA bypass)analysis by Neil MaddenCVE-2023-2728: Kubernetes mountable secrets policy bypassCVE-2023-2727: Kubernetes ImagePolicyWebhook bypassAuth0: Critical Vulnerabilities in JSON Web Token Libraries (alg:none, RS/HS confusion)PortSwigger Web Security Academy: JWT attacksjwt.io: Debugger and library reference

Tools and Projects – OAuth2 Proxy (GitHub — 14k+ stars)OAuth2 Proxy DocumentationGravitee.io API Gateway — JWT PolicyGravitee Kubernetes OperatorGravitee Helm ChartEmissary-Ingress AuthServiceOPA GatekeeperIstio Security: RequestAuthentication and AuthorizationPolicyEnvoy External Authorization FilterLinkerd Server PolicyOpen Policy AgentOPAL — Open Policy Administration LayerSPIFFE — Secure Production Identity Framework for EveryoneSPIRE — SPIFFE Runtime EnvironmentNetflix Zuul (GitHub)Traefik ForwardAuth Middleware

Architecture and Industry Context – Google BeyondCorp: A New Approach to Enterprise SecurityGoogle BeyondCorp Research Paper (USENIX ;login:)Netflix Tech Blog: Zuul 2 — The Netflix Journey to Asynchronous, Non-Blocking SystemsSPIFFE/SPIRE CNCF Graduation AnnouncementOPA CNCF GraduationInfoQ: Microservices Authentication and Authorization Anti-PatternsAWS re:Invent: Zero Trust Networking on AWSIstio Service Mesh Security ArchitectureThe CNCF TAG Security Microservices Security Paper


Article reflects tooling as of 2026: Kubernetes 1.29+, Istio 1.21+, Linkerd 2.15+, OPA 0.63+ / Gatekeeper 3.16+, SPIRE 1.9+, OAuth2 Proxy 7.6+, Gravitee APIM 4.x.

Sources

Kaniko, BuildKit, and Image Volumes: The Evolution of Container Images Inside Kubernetes

Kaniko, BuildKit, and Image Volumes: The Evolution of Container Images Inside Kubernetes

Running container images inside Kubernetes is table stakes. Building them there — or mounting their contents as volumes — has been a moving target for years. What started as a privileged hack has evolved into a set of mature, secure, and increasingly native primitives.

This article covers the full arc: why the original approach was broken, what Kaniko solved, why its maintenance status now matters, where BuildKit and Buildah fit, and what Image Volumes (stable in Kubernetes v1.36) change for workflows that don’t need to build anything at all.


The original problem: Docker-in-Docker

The first generation of CI/CD on Kubernetes ran Docker inside Docker. You mounted the host Docker socket (/var/run/docker.sock) into a build container, and that container had full access to the host’s Docker daemon.

# The approach nobody should be using in 2026
volumes:
- name: docker-sock
  hostPath:
    path: /var/run/docker.sock

This worked. It was also a complete security disaster.

Mounting the Docker socket gives the container root-equivalent access to the host. Any workload that can reach that socket can escape the container, inspect other containers, and compromise the node. The only thing standing between your CI pipeline and a full cluster compromise was the good intentions of whoever wrote the build script.

It got worse when the Kubernetes project deprecated Dockershim in v1.20 and removed it in v1.24. Clusters that moved to containerd or CRI-O no longer had a Docker daemon on the host at all. The socket either didn’t exist or belonged to a completely different runtime. Docker-in-Docker in its classic form became functionally impossible on modern clusters.


Kaniko: daemonless builds inside containers

Google released Kaniko in 2018 to solve exactly this problem. Kaniko builds container images entirely in userspace — no daemon, no privileged access, no host socket required.

The key insight: Docker builds work by executing each RUN instruction in a temporary container, snapshotting the filesystem, and saving the result as a layer. Kaniko replicates this logic without a daemon. It runs as a regular container, reads the Dockerfile, executes each step against the local filesystem, and pushes the resulting image directly to a registry.

That design aged well. The project governance did not. The GoogleContainerTools/kaniko repository was archived by its owner on June 3, 2025 and is now read-only. In practical terms, the original Google-hosted Kaniko project should be treated as unmaintained: no active upstream issue triage, no normal pull request flow, and no clear path for security fixes through that repository.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-build
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args:
    - "--dockerfile=Dockerfile"
    - "--context=git://github.com/your-org/your-repo"
    - "--destination=your-registry/your-image:tag"
    volumeMounts:
    - name: registry-creds
      mountPath: /kaniko/.docker
  volumes:
  - name: registry-creds
    secret:
      secretName: registry-credentials
      items:
      - key: .dockerconfigjson
        path: config.json
  restartPolicy: Never

No host socket. No privileged flag. The container needs write access to its own filesystem (so readOnlyRootFilesystem: true won’t work), but that’s a far narrower requirement than socket mounting.

When Kaniko is still a reasonable answer

Kaniko can still be a reasonable tactical choice when you need to build a container image inside Kubernetes and already have a working, isolated pipeline around it. Existing Kaniko jobs did not stop working when the repository was archived.

For new platform work in 2026, though, do not pick Kaniko by default. The maintenance signal changes the risk model. Use it only if the simplicity is worth owning the upgrade and vulnerability-management story yourself, or if you deliberately standardize on a maintained fork or vendor-supported distribution.

Kaniko integrates well with: – Tekton — the standard pattern is a Tekton Task running the Kaniko executor – Argo Workflows — same pattern, different orchestrator – GitLab CI on Kubernetes — many older examples and pipelines use Kaniko with the Kubernetes executor – Any pod-based CI system — it’s just a container, so it runs anywhere pods run


The alternatives: BuildKit and Buildah

Kaniko is no longer the only serious daemonless option. Two other tools are worth evaluating first for new work:

BuildKit (rootless mode)

BuildKit is the build backend behind docker buildx and the default build system in modern Docker. In rootless mode it runs without privileges and can build images inside a Kubernetes pod.

BuildKit has better caching than Kaniko — particularly layer caching via cache mounts — and supports more advanced Dockerfile features like heredocs and multi-platform builds. The tradeoff is more complex setup: you need to run buildkitd as a sidecar or as a DaemonSet.

For teams already using docker buildx locally, BuildKit is the most natural migration path. Docker’s Kubernetes driver can run BuildKit builders directly in a cluster, including rootless mode without privileged pods on supported Kubernetes versions. The operational cost is real — builder lifecycle, cache persistence, node placement, and rootless kernel requirements — but the project is active and the feature set is where most modern Dockerfile workflows are moving.

Buildah

Buildah is the containers project’s daemonless build tool, designed to integrate with Podman and OCI-native workflows. It can build from Dockerfiles or Containerfiles, and it is the natural choice on OpenShift or in environments where the platform team already standardizes on Red Hat, Podman, and the containers/* stack.

The caveat is that rootless Buildah inside a restricted Kubernetes pod still depends on user namespace behavior and helper binaries such as newuidmap and newgidmap. That is manageable on platforms built for it, especially OpenShift, but it is not automatically a drop-in replacement for every generic Kubernetes CI runner.


Image Volumes: a different problem entirely

Here is where the narrative splits. Everything above is about building images. Image Volumes solve a completely different problem: consuming the contents of an OCI image as a volume, without building anything.

The feature was introduced as alpha in Kubernetes v1.31, moved to beta in v1.33 with subPath and subPathExpr support, became beta enabled by default in v1.35, and graduated to stable (GA) in Kubernetes v1.36.

What it does

Image Volumes let you reference an OCI image in a pod’s volumes section and mount its filesystem contents directly into a container:

apiVersion: v1
kind: Pod
metadata:
  name: image-volume-example
spec:
  containers:
  - name: app
    image: debian
    command: ["sleep", "infinity"]
    volumeMounts:
    - name: config-data
      mountPath: /app/config
  volumes:
  - name: config-data
    image:
      reference: your-registry/your-config-image:v1.2.0
      pullPolicy: IfNotPresent

The container at /app/config sees the contents of your-config-image:v1.2.0. The volume is read-only. No init container required.

From Kubernetes v1.33+, you can also mount a subdirectory with subPath:

volumeMounts:
- name: config-data
  mountPath: /app/config
  subPath: environments/production

The use case: OCI images as artifact bundles

This feature is motivated by a pattern that has been growing quietly: using OCI images not as runnable containers but as versioned, signed, distributable artifact bundles.

The idea: instead of storing configuration, schemas, WASM modules, ML models, or static binaries in a ConfigMap or a separate volume, you package them as an OCI image. You get:

  • Version control — image tags and digests, same tooling you already use
  • Distribution — your existing registry, your existing pull secrets, your existing access controls
  • Signing and attestation — cosign, Sigstore, the full supply chain tooling works on these artifacts
  • Immutability — a digest-pinned image reference is cryptographically immutable

Image Volumes are the Kubernetes primitive that makes this pattern first-class. Without Image Volumes, the workaround was an init container that pulled the image and copied the contents to an emptyDir. It worked, but it was boilerplate.

What it does not do

Image Volumes are not a replacement for Kaniko or any build tool. They consume images; they don’t produce them. If your workflow involves building a new image from source, you still need Kaniko, BuildKit, or equivalent.

They also require container runtime support. CRI-O supported the initial alpha implementation from its v1.31 line and tracked beta support for v1.33; containerd support landed later through the containerd 2.x line. On anything before Kubernetes v1.36, verify both the Kubernetes feature gate and the runtime version before treating this as production plumbing.


Decision framework by Kubernetes version

K8s versionBuild images (CI)Mount image contents as volume
< 1.24Avoid Docker socket builds; use BuildKit, Buildah, or legacy Kaniko with explicit risk acceptanceInit container + emptyDir workaround
1.24 – 1.30BuildKit or Buildah preferred; legacy Kaniko only if already standardizedInit container + emptyDir workaround
1.31 – 1.32BuildKit or Buildah preferred; legacy Kaniko only with maintenance planImage Volumes alpha — ImageVolume feature gate required, runtime support required
1.33 – 1.34BuildKit or Buildah preferred; legacy Kaniko only with maintenance planImage Volumes beta with subPath/subPathExpr, but disabled by default; enable ImageVolume and verify runtime support
1.35BuildKit or Buildah preferred; legacy Kaniko only with maintenance planImage Volumes beta, enabled by default; still verify runtime support before production rollout
1.36+BuildKit or Buildah preferred; legacy Kaniko only for existing pipelines or maintained forksImage Volumes stable (GA), enabled by default

When to use what

Use Kaniko if: – You already have working Kaniko pipelines and the operational cost of migration is higher than the current risk – You have a maintained fork, vendor support, or an internal patching process – You want the simplest daemonless build setup and accept that upstream Google Kaniko is archived

Use BuildKit (rootless) if: – You need advanced cache mounts or multi-platform builds – Your team already uses docker buildx locally – You’re willing to run and operate BuildKit builders in the cluster

Use Buildah if: – You are on OpenShift or a Podman/Red Hat-oriented platform – You want an OCI-native build tool that does not require a daemon – Your cluster policy supports the user namespace requirements of rootless builds

Use Image Volumes if: – You want to inject versioned, signed artifacts into pods without building anything – You’re replacing init container + emptyDir patterns – You’re adopting OCI images as a general artifact format (configs, schemas, binaries)


Conclusion

The container image story inside Kubernetes has matured significantly. Docker-in-Docker is dead — correctly so. Kaniko solved an important build problem in 2018, but the original Google project is archived in 2026, so it should no longer be the default recommendation for new platforms. BuildKit and Buildah are the healthier starting points for active build pipelines, with Kaniko reserved for existing estates or explicitly supported forks.

Image Volumes are genuinely new ground. They’re not competing with Kaniko — they’re addressing a different layer of the same ecosystem: the distribution and consumption of OCI artifacts beyond just “images you run.” With GA in v1.36 and the supply chain tooling around OCI reaching maturity, the pattern of “package it as an image, distribute it like an image, mount it where you need it” is becoming the right answer for a class of problems that previously lived in ConfigMaps, PVCs, or init container hacks.

The right combination depends on what you’re doing and what version you’re running. But for the first time, Kubernetes has native answers for both sides of the equation.

Choose today

If you operate Kubernetes v1.36 or newer, use Image Volumes for read-only artifact injection and choose BuildKit or Buildah for builds. If you are on v1.35, Image Volumes are beta and enabled by default, but you should still verify runtime support and keep the init-container pattern as the rollback path. If you are on v1.33 or v1.34, beta does not mean default-on: enable the ImageVolume feature gate deliberately and validate the runtime. If you are on v1.31 or v1.32, treat Image Volumes as alpha and non-default. If you are below v1.31, Image Volumes are not part of the platform: use emptyDir plus an init container for artifact mounting, and modernize the build path separately.

For CI builds, start with BuildKit when you want Dockerfile compatibility, cache performance, multi-platform output, and a path aligned with docker buildx. Start with Buildah when your cluster is OpenShift or Podman-oriented. Keep Kaniko only where it already works and where someone owns the maintenance risk.

Sources

  • https://github.com/GoogleContainerTools/kaniko
  • https://github.com/kubernetes/enhancements/issues/4639
  • https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
  • https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-volume-source/
  • https://kubernetes.io/blog/2025/04/29/kubernetes-v1-33-image-volume-beta/
  • https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/
  • https://docs.docker.com/build/builders/drivers/kubernetes/
  • https://github.com/moby/buildkit/blob/master/docs/rootless.md
  • https://github.com/containers/buildah
  • https://github.com/containers/buildah/blob/main/docs/tutorials/05-openshift-rootless-build.md

EKS Auto Mode: What It Actually Changes (and What It Doesn’t)

EKS Auto Mode: What It Actually Changes (and What It Doesn't)

What EKS Auto Mode is

EKS Auto Mode, generally available on December 1, 2024, shifts more Kubernetes infrastructure responsibility from you to AWS. You still run an EKS cluster in your AWS account, and your workloads still use the Kubernetes API, but AWS takes over much of the compute, storage, networking, node lifecycle, and core add-on management that platform teams usually wire together themselves.

For compute, Auto Mode uses Karpenter-style provisioning under the hood. When pods are unschedulable, Auto Mode provisions nodes that fit the workload’s requirements: instance family, size, architecture, capacity type, and availability zone. When capacity is no longer useful, it can consolidate and terminate nodes.

The important framing is this: Auto Mode is not “EKS without nodes.” It is EKS where AWS manages the node lifecycle more aggressively. You own the workloads, their scheduling requirements, their disruption behavior, and the operational consequences of those choices. AWS owns more of the infrastructure plumbing.


What it replaces

Before Auto Mode, running EKS in production usually meant choosing and operating several layers yourself:

Managed node groups: You chose instance types, defined scaling ranges, managed AMI updates, handled node draining, and configured Cluster Autoscaler or another scaling mechanism.

Self-managed Karpenter: More flexible than managed node groups, but you owned the Karpenter controller, IAM, NodePools, EC2NodeClasses, disruption settings, upgrades, and failure modes.

Fargate: AWS-managed compute per pod, with no node management, but no DaemonSets, a narrower workload compatibility envelope, and a different cost model.

EKS Auto Mode replaces a large part of that platform assembly with a managed model: declare workload intent and high-level compute constraints; AWS provisions and manages the EC2 instances behind it.


How it works in practice

You create or update an EKS cluster with Auto Mode enabled. The default setup can use AWS-managed built-in node pools. If you need more control, you create a NodeClass for Auto Mode infrastructure settings and a Karpenter NodePool for workload-facing scheduling constraints.

# NodeClass: EKS Auto Mode infrastructure settings for managed EC2 nodes.
apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: private-compute
spec:
  subnetSelectorTerms:
    - tags:
        kubernetes.io/role/internal-elb: "1"
  securityGroupSelectorTerms:
    - tags:
        aws:eks:cluster-name: prod-eks
  ephemeralStorage:
    size: "100Gi"
# NodePool: workload-facing constraints for nodes that Auto Mode may provision.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: private-compute
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: eks.amazonaws.com/instance-category
          operator: In
          values: ["c", "m", "r"]
  limits:
    cpu: "1000"
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

Those API groups are the Auto Mode-specific split documented by AWS: apiVersion: eks.amazonaws.com/v1, kind: NodeClass for Auto Mode node infrastructure, and apiVersion: karpenter.sh/v1, kind: NodePool for scheduling and capacity constraints.

The NodeClass is where you express AWS infrastructure placement and node-level defaults. The NodePool is where you express what kind of capacity is acceptable for workloads. Do not copy a self-managed Karpenter EC2NodeClass into Auto Mode; Auto Mode uses its own NodeClass API.

Auto Mode provisions nodes when pods are pending, consolidates when nodes are underutilized, and replaces nodes during maintenance or scale-down. AMI and node lifecycle updates are handled by AWS. AWS says Auto Mode AMIs are generally released weekly with CVE and security fixes, and Auto Mode nodes have a maximum lifetime of 21 days, which you can reduce. Your application still has to tolerate the disruption: a bad PodDisruptionBudget, strict affinity rule, or singleton stateful workload can still block or degrade a replacement.

Built-in components are managed differently than in a classic EKS build. AWS lists pod networking, service networking, cluster DNS, autoscaling, block storage, load balancer controller, Pod Identity agent, and node monitoring agent as Auto Mode capabilities. With Auto Mode compute, common add-ons such as Amazon VPC CNI, kube-proxy, CoreDNS, Amazon EBS CSI Driver, and EKS Pod Identity Agent become redundant for Auto Mode nodes, and the relevant controllers can run on AWS-owned infrastructure rather than as visible pods in your account. You can still install AWS Load Balancer Controller in an Auto Mode cluster when you need both models during migration, but AWS does not support directly migrating existing load balancers from AWS Load Balancer Controller to Auto Mode; use IngressClass or loadBalancerClass boundaries and plan blue-green migration. Treat this as a change in ownership, not as a reason to skip validation.

The workload support matrix is broader than Fargate, but not identical to self-managed nodes:

CapabilityAuto Mode status
EC2 SpotSupported through karpenter.sh/capacity-type requirements such as spot, on-demand, and reserved
Graviton / arm64Supported through kubernetes.io/arch: arm64 and supported Graviton instance families
GPU / acceleratorsSupported for documented accelerated families; Auto Mode manages NVIDIA, Trainium, and Inferentia drivers/device plugins for supported instance types
Windows nodesNot supported
DaemonSetsSupported as Kubernetes DaemonSets, but host-level assumptions must be validated against locked-down managed nodes

What you gain

Reduced operational surface. Node group management, AMI lifecycle, Cluster Autoscaler tuning, Karpenter controller upgrades, and a chunk of add-on wiring move out of your day-to-day scope.

Better provisioning shape by default. Dynamic provisioning is usually a better fit than fixed node group shapes. You get nodes that more closely match actual pod requirements instead of trying to pre-plan a small set of instance types.

Automatic node patching. AWS manages the node image and replacement flow. That reduces toil, but it also means your workloads need disruption policies that let AWS replace nodes safely.

Faster cluster bootstrapping. A new Auto Mode cluster can get to a usable production baseline faster than a hand-assembled EKS cluster with node groups, autoscaling, networking add-ons, storage drivers, and load balancer controllers.

Native Spot integration. Auto Mode can use Spot capacity through NodePool requirements, but you still need workload-level interruption tolerance: replicas, budgets, graceful shutdown, and queue semantics where relevant.


What you give up

Node-level access. Auto Mode nodes are intentionally locked down compared with traditional self-managed nodes. If your incident response process assumes SSH, SSM, manual package inspection, or ad hoc host changes, it needs to change.

Custom AMIs. You cannot treat the node image as your own artifact. AWS determines the operating system and AMI for Auto Mode managed instances; you cannot directly access the instance or install software on it. If your organization requires internally built, hardened, or certified AMIs, Auto Mode is likely blocked.

Unrestricted host agents. Kubernetes DaemonSets are supported, but they are the sharp edge. Some node agents work; others do not. Anything that assumes privileged host access, custom kernel modules, hostPath writes, IMDS access without hostNetwork, or low-level runtime integration needs a proof of compatibility.

Less tuning surface. You give up direct control over kubelet flags, container runtime configuration, bootstrap scripts, and arbitrary node setup. That is the point of the product, but it is also the boundary.

Different cost visibility. Managed node groups make capacity easier to reason about because you chose it up front. Auto Mode changes capacity dynamically, so cost control moves toward budgets, labels, reports, and workload-level resource hygiene.


EKS Auto Mode vs managed node groups vs Fargate

Auto ModeManaged Node GroupsFargate
Node managementAWS manages node lifecycleShared: AWS manages the group primitive, you manage capacity shape and many updatesAWS-managed per-pod compute
AMI updatesAutomatic through AWS-managed node replacementYou schedule and operate rolling updatesN/A
Instance selectionDynamic through NodePoolsYou choose instance types and scaling rangesNot exposed
Custom AMIsNo; AWS determines the AMIYesNo
DaemonSetsSupported, but validate host-access assumptionsYesNo
SSH / node accessRestrictedUsually available if you enable itNo
Spot supportYes, through capacity-type requirementsYes, with node group or Karpenter designNo; Amazon EKS does not support Fargate Spot
Cost modelEKS control plane + EC2 + EKS Auto Mode feeEKS control plane + EC2EKS control plane + Fargate pod pricing
Operational burdenLow for nodes, medium for workload compatibilityMediumLow for nodes, medium for compatibility
Right forDefault candidate for teams that do not need node customizationRegulated/custom node environments and mature platform teamsWorkloads that fit Fargate’s restrictions and want per-pod isolation
Hidden constraints / gotchasAWS controls node image and lifecycle; DaemonSets, privileged pods, hostPath, PDBs, topology rules, and node agents can block migrationYou still own AMI drift, autoscaler tuning, disruption handling, and capacity fragmentationNo DaemonSets, limited host-level integrations, different networking/storage constraints, and less flexibility for mixed workload shapes

The Auto Mode pricing

Do not model Auto Mode as “EC2 plus a generic percentage” unless you have pulled the actual rate for your region and instance mix. The official structure is:

Total EKS Auto Mode cluster cost =
  EKS control plane cost
  + normal EC2 cost for instances launched and managed by Auto Mode
  + EKS Auto Mode management fee on those managed EC2 instances
  + normal surrounding AWS costs: EBS, load balancers, data transfer, CloudWatch, etc.

EKS Auto Mode management fee =
  sum of Auto Mode-managed instance runtime
  x the regional EKS Auto Mode management rate for each EC2 instance type

The EKS Auto Mode fee is applied to the EC2 instances that Auto Mode launches and manages. It is billed in addition to the normal EC2 charge and in addition to the EKS control plane charge. AWS bills the Auto Mode fee per second with a one-minute minimum, and the charge is independent of whether the underlying EC2 capacity is On-Demand, Spot, covered by Reserved Instances, or covered by Compute Savings Plans.

Do not treat the fee as a contractual flat percentage. The official pricing page describes it as a management fee that varies by EC2 instance type, and AWS pricing data is regional. The public pricing example for US West (Oregon) shows c6a.2xlarge at $0.306/hour for EC2 plus $0.03672/hour for Auto Mode, c6a.4xlarge at $0.612/hour plus $0.07344/hour, m5a.2xlarge at $0.344/hour plus $0.04128/hour, and m5a.xlarge at $0.172/hour plus $0.02064/hour. Those examples equal 12% of the listed On-Demand EC2 rate, but the safe formula for real planning is: sum(instance-hours by instance type and region x published Auto Mode management rate).

The practical cost question is not “is there a premium?” There is. The useful question is whether the premium is lower than the engineering time, incident risk, and opportunity cost of operating node lifecycle yourself.

For small teams, the answer may be yes even if the raw bill increases. For high-scale, cost-sensitive platforms, the answer needs real data: compare current EC2 waste, bin-packing efficiency, Spot usage, interruption rate, and platform maintenance time against an Auto Mode pilot.


When to use EKS Auto Mode

Use Auto Mode if:

  • You run EKS on AWS and do not have a hard requirement to manage nodes yourself
  • You do not require custom AMIs or custom node bootstrap logic
  • You want to reduce the operations surface for node lifecycle management
  • You want Karpenter-like provisioning without operating Karpenter yourself
  • Your workloads are mostly stateless or disruption-tolerant
  • Your observability, security, and storage agents are compatible with Auto Mode

Stick with managed node groups if:

  • Your organization requires internally certified or hardened AMIs
  • You need specific kernel configuration, kubelet flags, bootstrap scripts, or host packages
  • You depend on privileged DaemonSets or host-level security tooling that Auto Mode cannot support
  • You are in a regulated environment where the node image supply chain must be owned internally
  • Your platform team already operates Karpenter well and values the extra control

Use Fargate if:

  • You specifically want per-pod compute isolation
  • Your workload does not need DaemonSets or host-level integrations
  • You accept Fargate’s scheduling, networking, storage, and observability constraints
  • You want to avoid managing EC2 capacity entirely for a narrow class of workloads

Migration from managed node groups

Migrating an existing cluster to Auto Mode is supported, but it is not a one-command operational migration. AWS supports enabling Auto Mode on existing clusters, but you must update the cluster IAM role permissions and trust policy, enable compute, block storage, and load balancing capabilities together, and meet required add-on versions when those add-ons are installed. AWS also calls out unsupported direct migrations for EBS volumes from the standard EBS CSI provisioner to the Auto Mode EBS CSI provisioner, existing load balancers from AWS Load Balancer Controller to Auto Mode, and clusters using alternative CNIs or other unsupported networking configurations. A conservative path looks like this:

  1. Enable Auto Mode on a non-production cluster running Kubernetes 1.29 or greater.
  2. Inventory workloads by scheduling assumptions: node selectors, affinities, tolerations, topology spread, PDBs, privileged mode, hostPath, local storage, and DaemonSet dependencies.
  3. Create or select the relevant Auto Mode NodeClass and NodePool resources.
  4. Move a low-risk namespace first by changing selectors, tolerations, or labels so pods land on Auto Mode nodes.
  5. Watch scheduling, replacement, load balancer behavior, persistent volume provisioning, logging, metrics, and security events.
  6. Taint old node groups to stop new scheduling once the pilot workloads are stable.
  7. Drain old nodes gradually and delete old managed node groups only after workload owners have signed off.

The hardest part is usually not enabling Auto Mode. It is discovering which workloads and platform agents quietly depended on a mutable node.


What breaks when you migrate

Auto Mode changes the node contract. The Kubernetes API still looks familiar, but the host underneath is no longer yours in the same way.

DaemonSets need a compatibility audit. Logging agents, metrics agents, service mesh node components, security scanners, CSI node plugins, and custom infrastructure daemons often assume host access. Datadog, Falco, custom CSI drivers, eBPF agents, file integrity tools, and in-house node agents should be tested explicitly rather than assumed compatible.

PodDisruptionBudgets can block AWS-managed maintenance. If every critical Deployment has maxUnavailable: 0, or singleton workloads have no safe disruption path, node replacement becomes harder. Auto Mode can manage nodes, but it cannot make an application disruption-tolerant after the fact.

nodeSelector and affinity rules can strand pods. Workloads pinned to old node group labels, instance types, capacity labels, zones, or custom AMI labels may never schedule on Auto Mode capacity. Replace legacy labels with stable requirements that Auto Mode can satisfy.

topologySpreadConstraints can become too strict. Auto Mode provisions capacity dynamically, but strict zone spreading plus narrow selectors can create unschedulable pods. Check whenUnsatisfiable, label selectors, and minimum domain assumptions.

Privileged pods and hostPath volumes are migration blockers until proven otherwise. Anything that needs /var/lib, /proc, /sys, container runtime sockets, kernel capabilities, or host networking deserves a separate test. Some patterns are fundamentally at odds with locked-down managed nodes.

Observability and security agents may lose host assumptions. Agents that expect direct node access, host package installation, kernel modules, eBPF privileges, or container runtime socket access can fail partially. The dangerous failure mode is not “pod CrashLoopBackOff”; it is silent loss of telemetry or enforcement.

Storage drivers must be reviewed. EBS integration is part of Auto Mode, but it uses the Auto Mode EBS CSI provisioner ebs.csi.eks.amazonaws.com, not the standard EBS CSI provisioner ebs.csi.aws.com. Custom CSI drivers, EFS patterns, snapshot controllers, and topology-aware storage classes should be validated. Pay particular attention to provisioner names, volume binding mode, encryption settings, and IAM assumptions.

Runbooks need rewriting. “SSH to the node and inspect X” is not a valid first response anymore. Incident procedures should move toward kubectl describe, events, logs, ephemeral debug containers where supported, cloud-side metrics, and vendor-supported diagnostics.


The honest assessment

EKS Auto Mode is a good default candidate for many teams running Kubernetes on AWS. The operational simplification is real: node provisioning, AMI updates, core add-on integration, and scaling behavior are areas where teams burn time and create incidents.

The constraints are also real. Custom AMIs, unrestricted host access from DaemonSets, privileged pods, custom CSI drivers, and strict disruption policies are the common blockers. If your platform depends on those, Auto Mode is not a free upgrade.

For teams without those constraints, Auto Mode should be evaluated early for new EKS clusters. For existing clusters, it should be treated as a migration project, not a checkbox. The right question is not whether Auto Mode is “better” than managed node groups. The right question is which operational contract your workloads can actually live with.

The pattern is the same as with managed infrastructure generally: the more your organization can treat nodes as replaceable capacity, the more value you get. The more your platform treats nodes as customized machines, the less Auto Mode fits.


1-week pilot: evaluate Auto Mode without risking production

Use a short pilot to answer compatibility and economics questions before touching production.

  1. Create a test cluster or clone a representative non-production cluster. Use the same region, Kubernetes minor version, VPC shape, IAM model, ingress pattern, and storage classes where possible.
  2. Enable Auto Mode and deploy one custom NodeClass and NodePool. Keep the first pool boring: on-demand capacity, two or three common instance families, and the same private subnet pattern as production.
  3. Select three workload types. Pick one stateless service, one stateful service with EBS, and one platform-heavy workload that uses observability or security agents.
  4. Run a scheduling audit. Check node selectors, affinity, topology spread, PDBs, tolerations, privileged mode, hostPath, and DaemonSets before migration.
  5. Force normal failure modes. Roll deployments, delete pods, scale replicas up and down, trigger node consolidation if possible, and simulate one Spot-tolerant workload if you plan to use Spot.
  6. Validate platform signals. Confirm logs, metrics, traces, runtime alerts, security events, load balancer provisioning, DNS, and persistent volume operations.
  7. Compare costs and toil. Record EC2 instance mix, the published Auto Mode management fee for each instance type and region, pod density, pending time, interruption behavior, and operator actions required.

Success criteria should be explicit:

  • 95%+ of pilot pods schedule without manual intervention
  • No silent loss of logs, metrics, traces, or security alerts
  • PDBs allow node replacement for replicated services
  • Stateful workloads survive rescheduling and volume attachment tests
  • Cost model is understood at instance-family level, not estimated from a generic percentage
  • Production migration blockers are documented with owners

If the pilot fails, that is still useful. It tells you which node assumptions are real and which workloads should stay on managed node groups.


FAQ

Does EKS Auto Mode work with existing EKS clusters?

Yes, Auto Mode can be enabled on existing clusters running Kubernetes 1.29 or greater, provided the cluster meets the IAM, add-on, and networking requirements. Existing managed node groups can continue to run while you migrate workloads gradually. Treat mixed operation as a transition state with clear scheduling boundaries.

Can I still use kubectl and standard Kubernetes tooling with Auto Mode?

Yes. From the workload API perspective, it is still Kubernetes. kubectl, Helm, Argo CD, Flux, policy engines, and CI/CD workflows should continue to work unless they depend on node-level implementation details.

What happens when a node AMI has a CVE?

AWS manages the node image and replacement flow for Auto Mode nodes. Your responsibility is to make sure workloads can be disrupted safely: replicas, PDBs, graceful shutdown, readiness probes, and topology rules all matter.

Can I use my existing Karpenter NodePools and EC2NodeClasses?

Not directly. Auto Mode uses Karpenter NodePool resources, but the AWS-specific node class is NodeClass under eks.amazonaws.com/v1, not the self-managed Karpenter EC2NodeClass. Review every field before porting anything.

Is EKS Auto Mode available in all AWS regions?

At launch, AWS announced Auto Mode in all AWS Regions where EKS was available except AWS GovCloud (US) and China Regions. That is no longer the full current picture: AWS later announced availability in both AWS GovCloud (US-East) and AWS GovCloud (US-West), and AWS China announced availability in the China (Beijing) and China (Ningxia) Regions. AWS documentation also lists Auto Mode AMI accounts across current commercial and GovCloud Regions. Still verify the target Region before rollout, because regional launches and partition-specific requirements can lag; AWS China, for example, documents Kubernetes 1.30 or later for Auto Mode.

Does Auto Mode support Windows nodes?

No. AWS currently states that EKS Auto Mode does not support Windows nodes. Windows workloads should stay on managed node groups or self-managed Windows nodes.

Does Auto Mode remove the need for HPA or KEDA?

No. Auto Mode handles node provisioning and lifecycle. It does not decide how many replicas your application should run. You still need HPA, KEDA, custom controllers, or application-level scaling logic for pod replica counts.

Is Auto Mode cheaper than managed node groups?

Not automatically. Auto Mode adds a management fee on top of EC2 and EKS control plane costs. It may still lower total cost if it improves bin packing, reduces over-provisioning, increases Spot usage safely, or saves meaningful platform engineering time. Measure it with your workload mix.

What is the biggest migration risk?

Hidden node assumptions. DaemonSets, privileged pods, hostPath, strict PDBs, old node labels, custom CSI drivers, and security agents are the areas most likely to break or degrade silently.


Sources

Kubernetes Security Best Practices: 2026 Production Hardening Guide

Kubernetes Security Best Practices: 2026 Production Hardening Guide

Kubernetes security is not a single feature you enable — it is a layered discipline that spans the control plane, workloads, networking, supply chain, and runtime. Get one layer wrong and the others rarely save you. This guide covers the controls that matter most in production, why each one exists, and how to implement them without breaking your cluster — plus a prioritized roadmap so you know what to do in your first week, not just an undifferentiated list of “best practices.”

Related reading: the authentication-inside-containers anti-pattern.

Let me start with the part most hardening guides skip: what an actual attack looks like.

Anatomy of a Real Kubernetes Attack Chain

Abstract advice (“apply least privilege”) doesn’t land until you’ve seen how a single misconfiguration cascades. Here is a realistic chain — every step maps to a documented technique in the MITRE ATT&CK for Containers matrix. If you haven’t seen it before, ATT&CK is an industry-standard, openly maintained knowledge base of real-world adversary behavior: a catalogue of how attackers actually operate, organized by goal (initial access, credential access, lateral movement, and so on). It’s the common language security teams use to describe and defend against attacks.

  1. Initial access. An application pod runs a vulnerable image — say, an unpatched dependency with a remote code execution (RCE) flaw, a bug that lets an attacker run arbitrary code on the host process. The attacker gets code execution inside the container. So far, container isolation should contain the blast radius.
  2. Credential access. The pod has automountServiceAccountToken: true (the default). The attacker reads /var/run/secrets/kubernetes.io/serviceaccount/token — a valid API credential, handed to them for free.
  3. Discovery. Using that token, the attacker queries the API server. The ServiceAccount was bound to a convenient cluster-admin role “to unblock a deploy.” Now they can list every Secret in every namespace.
  4. Lateral movement. They read database credentials, cloud provider keys, and other ServiceAccount tokens from Secrets. The flat pod network (no NetworkPolicies) lets them reach internal services directly.
  5. Privilege escalation / escape. They schedule a privileged pod with hostPID and the host filesystem mounted, then break out to the node. From the node, they reach the kubelet and other tenants’ workloads.
  6. Impact. Crypto-mining, data exfiltration, or ransomware across the cluster.

Notice that steps 2 through 5 each had a one-line fix: disable token automount, scope the RBAC, encrypt Secrets / use an external store, apply default-deny NetworkPolicies, enforce Pod Security. Defense in depth means an attacker has to defeat every layer — and most attackers give up when the easy chain breaks. The rest of this guide is those layers, ordered by how much they shrink that chain.

The Kubernetes Attack Surface

Before hardening anything, understand what you are protecting. A Kubernetes cluster has several distinct attack surfaces:

  • API server — The central control plane. Any entity that can reach it with valid credentials can read cluster state, modify workloads, or escalate privileges.
  • etcd — Stores all cluster state in plain text, including Secrets. Direct etcd access is equivalent to root on every node.
  • Nodes — A compromised node can access all Secrets mounted on pods running on it, access the kubelet API, and potentially escape to the hypervisor.
  • Pods — Privileged pods, host-network pods, and pods with excessive capabilities can break container isolation.
  • Supply chain — Malicious images, compromised registries, and unsigned artifacts can introduce attacker-controlled code into your cluster.
  • RBAC — Overly permissive roles allow lateral movement and privilege escalation once an attacker gains any foothold.

Prioritize based on your threat model — a public-facing multi-tenant cluster needs all of these; an internal development cluster can relax some.

The First-Week Hardening Roadmap (Prioritized)

If you inherited a cluster with nothing in place, do not try to do everything at once. Order matters — some controls give huge risk reduction for minimal effort and zero breakage risk, others need careful rollout. This is the sequence I use:

DayControlRisk reductionBreakage risk
1Audit RBAC, remove stray cluster-admin, disable unused SA token automountHighLow
1Enable API server audit loggingMedium (visibility)None
2Pod Security Admission in warn + audit mode (all namespaces)HighNone (warn only)
3Deploy image scanning in CI (Trivy/Grype), fail on CriticalHighLow
4NetworkPolicies in audit-style rollout: default-deny in one namespace firstHighMedium — test DNS!
5Enable etcd encryption at rest / move Secrets to external storeHighLow
6Flip Pod Security Admission to enforce: baseline, then restricted per namespaceHighMedium
7Deploy runtime detection (Falco) + continuous scanning (Trivy Operator)MediumNone

The single most important idea: roll out enforcing controls in observation mode first (warn/audit for Pod Security, default-deny NetworkPolicies in one namespace). You want to discover what breaks in a dashboard, not in an incident.

Tools to automate and report each step

You don’t have to do any of this by hand. Each step has tooling that both applies the control and reports on its state, so you can wire it into CI or a recurring job:

1. RBAC: Least Privilege from Day One

Role-Based Access Control is Kubernetes’ primary authorization mechanism. Most clusters fail at RBAC not because it is misconfigured, but because it is over-permissive by default and nobody reviews it systematically.

Common RBAC Mistakes

  • Binding to cluster-admin for convenience. Almost no workload needs cluster-admin. Use namespaced roles wherever possible.
  • Using * verbs or resources in roles. Wildcard permissions are almost always broader than intended.
  • Not auditing ServiceAccount token usage. Every pod gets a ServiceAccount. Custom workloads often get over-permissive SAs.
  • Forgetting automountServiceAccountToken: false. If a workload does not need to talk to the Kubernetes API, disable token mounting entirely — this single setting breaks step 2 of the attack chain above.

Practical RBAC Patterns

For a workload that only needs to read ConfigMaps in its own namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: configmap-reader
  namespace: my-app
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-app-configmap-reader
  namespace: my-app
subjects:
- kind: ServiceAccount
  name: my-app
  namespace: my-app
roleRef:
  kind: Role
  name: configmap-reader
  apiGroup: rbac.authorization.k8s.io

And disable token automount on the workload that doesn’t call the API at all:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: my-app
automountServiceAccountToken: false

Audit existing RBAC with kubectl-who-can or rbac-tool to find overly permissive bindings before attackers do. A useful one-liner: list every subject that can read Secrets cluster-wide with kubectl who-can get secrets.

2. Pod Security Standards (and Migrating off PodSecurityPolicy)

PodSecurityPolicy was deprecated in Kubernetes 1.21 and removed in 1.25. Its replacement is Pod Security Admission (PSA), a built-in admission controller that enforces one of three Pod Security Standards profiles at the namespace level:

  • Privileged — No restrictions. For system components only.
  • Baseline — Prevents the most critical privilege escalations: privileged containers, hostPID, hostIPC, hostNetwork, dangerous capabilities.
  • Restricted — Enforces current hardening best practices. Requires running as non-root, dropping all capabilities, and using a restricted seccomp profile.

Enable enforcement at the namespace level with labels:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.30
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.30
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.30

A pod that runs as root or requests host-network in a namespace enforcing restricted will be rejected at admission. The warn and audit modes let you test before enforcing. For a full walkthrough of how PSA evaluates pods and how to roll it out, see my guide on understanding Pod Security Admission.

Migrating from PodSecurityPolicy to PSA

If you’re still on a cluster that used PSP, the migration path is:

  1. Map your PSPs to the closest PSA level. Most “restricted” PSPs map to restricted; permissive ones to baseline. The official pspmigrator tool can suggest mappings.
  2. Label every namespace in warn/audit mode matching that level — no enforcement yet.
  3. Watch the audit logs and warnings for a release cycle. Fix the workloads that would be rejected (add securityContext, drop capabilities).
  4. Flip to enforce namespace by namespace, starting with the least critical.

PSA is intentionally coarse-grained — three levels, namespace-scoped. For anything finer (per-team registries, required labels, custom mutation), you need a policy engine, which is the next section.

3. Policy Engines: Kyverno vs OPA Gatekeeper

Once you outgrow PSA’s three levels, you need an admission policy engine. The two standards are Kyverno and OPA Gatekeeper, and choosing between them is one of the most common platform decisions.

KyvernoOPA Gatekeeper
Policy languageYAML (Kubernetes-native)Rego (purpose-built DSL)
Learning curveLow — looks like other manifestsSteep — Rego is its own paradigm
Mutation supportYes, first-classLimited
Image verification (Cosign)Built-inVia external data
Best whenTeam wants fast adoption, K8s-onlyTeam already runs OPA across the stack

For most teams without existing Rego expertise, Kyverno is significantly faster to adopt and maintain. A Kyverno policy to require all images come from your private registry:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-registries
    match:
      any:
      - resources:
          kinds: ["Pod"]
    validate:
      message: "Images must come from registry.company.com"
      pattern:
        spec:
          containers:
          - image: "registry.company.com/*"

Both integrate cleanly with GitOps — store policies in Git, apply via Argo CD or Flux, and you get an auditable history of every policy change. I’ve written several deep dives on this: Kyverno: enforcing standard and custom policies, extending Kyverno with custom rules, and running the Kyverno CLI in CI/CD with GitHub Actions — or browse everything under the policies tag.

4. Network Policies: Micro-Segmentation

By default, every pod in a Kubernetes cluster can communicate with every other pod across all namespaces. This flat network model gives attackers unrestricted lateral movement once they compromise any workload (step 4 of the attack chain).

Network Policies define L3/L4 allow-rules for pod-to-pod communication. They are enforced by your CNI (Container Network Interface) plugin (Calico, Cilium, Weave — not Flannel, which does not support NetworkPolicy).

Default Deny Pattern

Start by denying all ingress and egress in a namespace, then open only what is explicitly needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Then allow specific traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - protocol: TCP
      port: 5432

The DNS Trap (the #1 reason default-deny “breaks everything”)

The most common NetworkPolicy support ticket: “I applied default-deny and the whole namespace stopped working.” The cause is almost always DNS. A default-deny egress policy blocks the pod from reaching kube-dns, so every name resolution fails and applications appear to hang or crash-loop.

Always pair default-deny egress with an explicit DNS allow rule:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Roll default-deny out in one namespace first, confirm DNS and required egress work, then expand. Tools like Cilium’s Hubble or Calico’s flow logs make it much easier to see exactly which flows you need to allow.

5. Secrets Management

Kubernetes Secrets are base64-encoded, not encrypted. They are stored in etcd in plain text by default. Anyone with get permission on Secrets can read them. This is not a vulnerability — it is a design decision that puts the responsibility on you to:

  • Enable encryption at rest for etcd. Configure EncryptionConfiguration with an AES-CBC or AES-GCM provider so Secrets are encrypted before being written to etcd.
  • Use external secret stores. HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault with the External Secrets Operator means actual secret values never live in Kubernetes at all.
  • Restrict Secret RBAC aggressively. Never give list on Secrets cluster-wide — it returns all values. Use get on named resources where possible.
  • Avoid environment variables for secrets. Prefer volume mounts. Env vars are visible in pod inspect output and can leak through application logging.
# etcd encryption at rest - in kube-apiserver config
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-32-byte-key>
  - identity: {}

For the full external-store pattern, see the guide on injecting secrets into pods with HashiCorp Vault.

6. Image Security and Supply Chain

Your runtime security posture is only as good as the images you run. A compromised image from a public registry bypasses every runtime control you have.

Scan images in CI

Use Trivy, Grype, or Snyk to scan images as part of CI. Block deployments of images with critical CVEs (Common Vulnerabilities and Exposures — publicly catalogued security flaws). I’ve covered the practical side of this in scanning Docker images with Trivy, scanning your images locally before they ship, and a broader roundup of open-source development security tools.

# In your CI pipeline
trivy image --exit-code 1 --severity CRITICAL your-image:tag

Use a private registry with admission control

Only allow images from your private registry using an admission webhook (Kyverno, OPA Gatekeeper) — the policy in section 3 does exactly this. It prevents developers from running arbitrary public images in production.

Use distroless or minimal base images

Distroless images contain only the application and its runtime dependencies — no shell, no package manager, no debugging tools. This drastically reduces both the attack surface and the CVE count. Google’s distroless images are available for Java, Node.js, Python, and Go. (Related: debugging distroless containers when you do need to inspect one.)

Sign and verify images (and the SLSA angle)

Cosign (from the Sigstore project) lets you sign container images and verify signatures at admission time using Kyverno or Connaisseur. This prevents image-substitution attacks where an attacker replaces a legitimate image in your registry.

If you’re being asked about supply-chain compliance, the framework to know is SLSA (Supply-chain Levels for Software Artifacts). The practical progression: SLSA L1 = you have a build provenance document; L2 = it’s signed and the build is hosted; L3 = the build is hardened and non-falsifiable. Generating provenance with your CI (GitHub Actions has native SLSA generators) and verifying it at admission with Cosign + Kyverno gets you most of the way to L2/L3 without a platform rebuild.

7. Runtime Security

Runtime security detects and responds to malicious activity after a container is running. The primary tool is Falco — a CNCF project that uses eBPF (extended Berkeley Packet Filter — a Linux kernel technology for running sandboxed observability programs) to monitor system calls and raise alerts when containers behave unexpectedly.

Default Falco rules catch common attack patterns:

  • Shell spawned in a container
  • Network connection to an unexpected IP
  • Write to a sensitive file path (/etc/passwd, /etc/shadow)
  • Privilege escalation via setuid binaries
  • Container drift (new executable files written at runtime)

Combine Falco with seccomp profiles to restrict the system calls a container can make at the kernel level. The RuntimeDefault seccomp profile (a default option since Kubernetes 1.27) blocks 300+ system calls that containers virtually never need.

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 65534
      capabilities:
        drop: ["ALL"]

These four securityContext settings together (allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, runAsNonRoot: true, capabilities.drop: ALL) make container escape significantly harder and satisfy the Kubernetes Restricted pod security standard. They directly close step 5 of the attack chain.

8. API Server Hardening

The API server is the most critical component to harden. Key settings:

  • Disable anonymous authentication. --anonymous-auth=false ensures every request is authenticated.
  • Enable audit logging. Log all API server requests to a file or webhook. Without audit logs, you cannot investigate incidents or detect RBAC abuse.
  • Restrict admission plugins. Ensure NodeRestriction is enabled — it prevents node kubelets from modifying objects outside their own node.
  • Do not expose the API server to the internet. Use a VPN, bastion host, or private endpoint. If you must expose it, restrict access by IP.
# Minimal audit policy - log all requests at metadata level,
# and full request body for sensitive resources
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets", "configmaps"]
- level: Metadata
  omitStages: ["RequestReceived"]

9. etcd Security

etcd stores all cluster state. Treat it as sensitive as your production database:

  • Enable TLS for all etcd communication — both peer (etcd-to-etcd) and client (apiserver-to-etcd) with mutual TLS.
  • Restrict network access to etcd. It should only be reachable by the API server. Use firewall rules or security groups.
  • Enable encryption at rest (see Secrets section).
  • Back up etcd regularly. A snapshot is a complete copy of all cluster state, including all Secrets. Encrypt backups and store them separately from the cluster.

10. Multi-Tenancy Isolation

If multiple teams or customers share a cluster, namespace boundaries alone are not a security boundary — they’re an organizational one. Hardening multi-tenant clusters adds requirements on top of everything above:

  • Namespace-per-tenant with ResourceQuotas and LimitRanges to prevent noisy-neighbor and resource-exhaustion DoS.
  • NetworkPolicies that deny cross-namespace traffic by default, so tenant A cannot reach tenant B’s pods.
  • A policy engine enforcing per-tenant rules (allowed registries, required labels, no hostPath).
  • Separate node pools for untrusted workloads, or a sandboxed runtime (gVisor, Kata Containers) when you run genuinely untrusted code.

For hard multi-tenancy (untrusted tenants), the honest answer is that vanilla namespaces aren’t enough — consider virtual clusters (vCluster) or separate clusters entirely. Soft multi-tenancy (trusted internal teams) is well served by the controls in this guide.

11. Benchmarks and Continuous Posture

CIS Kubernetes Benchmark

The CIS Kubernetes Benchmark is a comprehensive checklist covering the control plane, nodes, and workloads. Running kube-bench gives you a scored assessment:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs $(kubectl get pods -l app=kube-bench -o name)

kube-bench outputs PASS/FAIL/WARN for each control with remediation guidance. Run it after initial cluster setup and after major configuration changes.

Continuous scanning with Trivy Operator / Kubescape

Kubescape and the Trivy Operator provide continuous security scanning of live cluster state — not just a one-time audit. They check workloads against NSA/CISA hardening guidelines, the MITRE ATT&CK framework, and the CIS benchmark in real time.

helm repo add aquasecurity https://aquasecurity.github.io/helm-charts/
helm install trivy-operator aquasecurity/trivy-operator 
  --namespace trivy-system 
  --create-namespace 
  --set="trivy.ignoreUnfixed=true"

Trivy Operator creates VulnerabilityReport, ConfigAuditReport, and RbacAssessmentReport custom resources alongside each workload. Scrape them with Prometheus and build a security dashboard in Grafana.

Security Hardening Checklist

  • ✅ RBAC reviewed — no wildcard roles, no unnecessary cluster-admin bindings
  • ✅ ServiceAccount token automount disabled for workloads that do not need API access
  • ✅ Pod Security Standards enforced at namespace level (at least Baseline, Restricted where possible)
  • ✅ Policy engine (Kyverno/Gatekeeper) enforcing registry, label, and mutation rules
  • ✅ Network policies deployed — default deny with explicit allows (including DNS!)
  • ✅ Secrets encrypted at rest in etcd or moved to an external store
  • ✅ Images scanned in CI — no critical CVEs in production
  • ✅ Private registry enforced via admission control
  • ✅ Image signing + verification (Cosign) and build provenance (SLSA)
  • ✅ Container securityContext hardened (non-root, read-only fs, no capabilities)
  • ✅ seccomp RuntimeDefault profile enabled
  • ✅ API server audit logging enabled, anonymous auth disabled
  • ✅ etcd TLS and network access restricted
  • ✅ Multi-tenancy isolation (quotas, cross-namespace deny) if shared
  • ✅ kube-bench run and critical/high findings remediated
  • ✅ Runtime security (Falco) deployed and alerts routed to on-call
  • ✅ Continuous scanning (Trivy Operator or Kubescape) deployed

FAQ

Where do I start if my cluster has no security controls today?

Follow the first-week roadmap above. The short version: audit RBAC (revoke stray cluster-admin), enable Pod Security Admission in warn mode on all namespaces, and deploy image scanning + Trivy Operator. These give immediate visibility and stop the most common privilege escalations without breaking anything.

Does enabling Network Policies break DNS resolution?

Yes — this is the single most common failure. A default-deny egress policy blocks pods from reaching kube-dns, so name resolution fails. Add an egress rule allowing UDP and TCP port 53 to the kube-system namespace whenever you apply default-deny (see the DNS allow policy above).

Should I use OPA Gatekeeper or Kyverno?

Both enforce admission policies. Kyverno is Kubernetes-native (policies are YAML) while Gatekeeper uses Rego. For teams without Rego expertise, Kyverno is faster to adopt and supports mutation and Cosign verification out of the box. Choose Gatekeeper if you already run OPA elsewhere and want one policy language across your stack.

What replaced PodSecurityPolicy?

Pod Security Admission (PSA), built into Kubernetes since 1.25. It enforces three profiles (privileged/baseline/restricted) via namespace labels. For finer-grained control than PSA’s three levels, add Kyverno or Gatekeeper.

Is Kubernetes certified for PCI-DSS or SOC 2?

Kubernetes itself is not certified — your configuration and the controls you implement determine compliance. The CIS Kubernetes Benchmark maps to many PCI-DSS and SOC 2 requirements. Managed offerings (EKS, GKE, AKS) carry their own compliance certifications for the underlying infrastructure.

How often should I update Kubernetes for security patches?

Apply a patch release within 30 days for High/Critical CVEs. Minor version upgrades (e.g., 1.30 → 1.31) should happen within the support window — Kubernetes maintains the last three minor versions. Falling more than one minor behind means running without patches for a growing subset of the codebase.

Are namespaces a security boundary?

No. Namespaces are an organizational boundary. Real isolation between tenants requires NetworkPolicies, ResourceQuotas, a policy engine, and — for untrusted workloads — sandboxed runtimes (gVisor/Kata) or separate clusters.


For a deeper look at how security fits into the broader Kubernetes platform architecture, see the Kubernetes architecture patterns guide and the guide on building a security-first Kubernetes culture.

ArgoCD Guide: GitOps Continuous Delivery for Kubernetes

ArgoCD Guide: GitOps Continuous Delivery for Kubernetes

ArgoCD has become the de facto standard for GitOps-based continuous delivery in Kubernetes. If you are running production workloads on Kubernetes and still deploying with raw kubectl apply or untracked Helm releases, ArgoCD solves a class of problems you may not even know you have yet. This guide covers everything from core concepts to production-grade configuration.

The Problem ArgoCD Solves

Traditional CI/CD pushes deployments into a cluster. A CI system runs tests, builds an image, and then executes kubectl apply or helm upgrade against the cluster. This model has several structural problems:

  • Drift goes undetected. Someone applies a hotfix directly to the cluster. Now your Git repository no longer reflects reality, and nobody knows it.
  • No single source of truth. The cluster state is authoritative, not Git. Your desired state and actual state can diverge silently.
  • Rollback is painful. Rolling back a bad deployment means re-running old CI pipelines or manually reversing changes, neither of which is fast.
  • Multi-cluster management compounds the problem. Each cluster becomes a snowflake with its own history of undocumented changes.

GitOps inverts this model. Git is the source of truth. The cluster pulls its desired state from Git and continuously reconciles toward it. ArgoCD is the most mature GitOps operator for Kubernetes, implementing this pull-based model with a production-ready feature set.

How ArgoCD Works: Core Architecture

ArgoCD runs as a set of controllers inside your Kubernetes cluster. The core components are:

  • Application Controller — Watches both the Git repository and the live cluster state. Computes the diff and drives reconciliation.
  • API Server — Exposes the gRPC/REST API consumed by the CLI, UI, and external systems.
  • Repository Server — Generates Kubernetes manifests from source (Helm, Kustomize, plain YAML, Jsonnet).
  • Redis — Caches cluster state and repository data to reduce API server load.
  • Dex (optional) — Provides OIDC authentication for SSO integration.

The fundamental unit in ArgoCD is an Application — a CRD that maps a source (a path in a Git repo at a specific revision) to a destination (a namespace in a cluster). ArgoCD continuously compares the desired state from Git with the live state in the cluster and reports on the sync status.

Sync Status vs Health Status

Two orthogonal concepts you need to understand from day one:

  • Sync Status — Does the live state match what Git says it should be? Values: Synced, OutOfSync, Unknown.
  • Health Status — Is the application actually working? Values: Healthy, Progressing, Degraded, Suspended, Missing, Unknown.

An application can be Synced but Degraded — the manifests were applied correctly, but a pod is crash-looping. Conversely, it can be OutOfSync but Healthy — someone applied a change directly to the cluster outside of Git.

Installing ArgoCD

The official installation method uses a single manifest. For production, always pin to a specific version:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.11.0/manifests/install.yaml

This deploys ArgoCD in the argocd namespace with full cluster-admin access. For a production HA setup, use the manifests/ha/install.yaml variant, which deploys multiple replicas of the API server and application controller.

Accessing the UI and CLI

The initial admin password is auto-generated and stored in a secret:

argocd admin initial-password -n argocd

For local access, port-forward the API server:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Then log in via the CLI:

argocd login localhost:8080 --username admin --password <password> --insecure

For production, expose the ArgoCD server via an Ingress or LoadBalancer with a proper TLS certificate. If you’re using NGINX Ingress Controller:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  rules:
  - host: argocd.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              number: 443

Defining Your First Application

Applications can be created via the UI, the CLI, or declaratively with a YAML manifest. The declarative approach is the recommended one — it means your ArgoCD configuration itself is in Git:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/your-app
    targetRevision: HEAD
    path: k8s/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Key fields to understand:

  • targetRevision — Can be a branch name, tag, or commit SHA. For production, pin to a tag rather than HEAD.
  • path — The directory within the repo containing your Kubernetes manifests.
  • automated.prune — Automatically delete resources that are no longer in Git. Required for true GitOps but use carefully — it will delete things.
  • automated.selfHeal — Automatically revert manual changes made directly to the cluster. This is what enforces Git as the single source of truth.

Helm Integration

ArgoCD has native Helm support. It can deploy Helm charts directly from chart repositories or from your Git repository. You can override values per environment:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prometheus-stack
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    chart: kube-prometheus-stack
    targetRevision: 58.4.0
    helm:
      releaseName: prometheus-stack
      valuesObject:
        grafana:
          adminPassword: "${GRAFANA_PASSWORD}"
        prometheus:
          prometheusSpec:
            retention: 30d
            storageSpec:
              volumeClaimTemplate:
                spec:
                  storageClassName: fast-ssd
                  resources:
                    requests:
                      storage: 50Gi
  destination:
    server: https://kubernetes.default.svc
    namespace: observability

One important nuance: ArgoCD renders Helm charts server-side using its own templating engine, not helm install. This means Helm hooks (pre-install, post-upgrade, etc.) are supported, but the release is not tracked in Helm’s release history. Running helm list will not show ArgoCD-managed releases unless you configure ArgoCD to use the Helm secrets backend.

Projects: Multi-Tenancy and Access Control

ArgoCD Projects provide multi-tenancy within a single ArgoCD instance. They let you restrict which source repositories, destination clusters, and namespaces a team can deploy to. Every Application belongs to a Project.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform-team
  namespace: argocd
spec:
  description: Platform team applications
  sourceRepos:
  - 'https://github.com/your-org/*'
  destinations:
  - namespace: 'platform-*'
    server: https://kubernetes.default.svc
  clusterResourceWhitelist:
  - group: ''
    kind: Namespace
  namespaceResourceBlacklist:
  - group: ''
    kind: ResourceQuota

Projects are where you define the boundaries of what each team can do. The default project has no restrictions — never use it for production workloads. Create dedicated projects per team or per environment.

RBAC Configuration

ArgoCD has its own RBAC system layered on top of Kubernetes RBAC. It is configured via the argocd-rbac-cm ConfigMap. Roles are defined per project or globally:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Platform team has full access to platform-team project
    p, role:platform-admin, applications, *, platform-team/*, allow
    p, role:platform-admin, projects, get, platform-team, allow
    p, role:platform-admin, repositories, *, *, allow

    # Dev team can sync but not delete
    p, role:developer, applications, get, */*, allow
    p, role:developer, applications, sync, */*, allow
    p, role:developer, applications, action/*, */*, allow

    # Bind SSO groups to roles
    g, your-org:platform-team, role:platform-admin
    g, your-org:developers, role:developer

The policy.default: role:readonly ensures that any authenticated user who has no explicit role assignment gets read-only access — a safe default for production.

Multi-Cluster Management

ArgoCD can manage multiple Kubernetes clusters from a single control plane. Register external clusters with the CLI:

# First, ensure the target cluster context is in your kubeconfig
argocd cluster add production-eu-west --name production-eu-west

# Verify registration
argocd cluster list

ArgoCD will create a ServiceAccount in the target cluster and store its credentials as a Kubernetes secret in the ArgoCD namespace. Applications can then target this cluster by name in their destination.server field.

For large-scale multi-cluster setups, consider the App of Apps pattern or ApplicationSets. ApplicationSets are a controller that generates Applications dynamically based on generators — cluster lists, Git directory structures, or matrix combinations:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-addons
  namespace: argocd
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          environment: production
  template:
    metadata:
      name: '{{name}}-addons'
    spec:
      project: platform
      source:
        repoURL: https://github.com/your-org/cluster-addons
        targetRevision: HEAD
        path: 'addons/{{metadata.labels.region}}'
      destination:
        server: '{{server}}'
        namespace: kube-system

This single ApplicationSet deploys the appropriate addons to every cluster labeled environment: production, using each cluster’s region label to select the correct path in the repository.

Sync Strategies and Waves

When deploying complex applications with dependencies between resources, you need to control the order of deployment. ArgoCD provides two mechanisms:

Sync Phases

Resources are deployed in three phases: PreSync, Sync, and PostSync. Use Sync Hooks for resources that must complete before the main sync proceeds (database migrations, certificate issuance, etc.):

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: your-app:v1.2.3
        command: ["./migrate.sh"]
      restartPolicy: Never

Sync Waves

Within the Sync phase, waves control ordering. Resources with a lower wave number are applied and must become healthy before resources with higher wave numbers are applied:

# Applied first
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

# Applied after wave 1 is healthy
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"

Notifications and Alerting

ArgoCD Notifications is a standalone controller that sends alerts when Application state changes. It supports Slack, PagerDuty, GitHub commit status, email, and a dozen other providers. Configure it via the argocd-notifications-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token
  template.app-sync-failed: |
    slack:
      attachments: |
        [{
          "title": "{{.app.metadata.name}}",
          "color": "#E96D76",
          "fields": [{
            "title": "Sync Status",
            "value": "{{.app.status.sync.status}}",
            "short": true
          },{
            "title": "Message",
            "value": "{{range .app.status.conditions}}{{.message}}{{end}}",
            "short": false
          }]
        }]
  trigger.on-sync-failed: |
    - when: app.status.sync.status == 'Unknown'
      send: [app-sync-failed]
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed]

Secret Management with ArgoCD

ArgoCD intentionally has no secret management built in — storing secrets in Git as plain text is never acceptable. The common patterns are:

  • Sealed Secrets (Bitnami) — Encrypts secrets with a cluster-specific key. The encrypted secret can be committed to Git; only the cluster can decrypt it.
  • External Secrets Operator — Syncs secrets from Vault, AWS Secrets Manager, GCP Secret Manager, etc. into Kubernetes secrets. The ArgoCD Application manages the ExternalSecret CRD, not the actual secret value.
  • argocd-vault-plugin — A plugin that replaces placeholder values in manifests with secrets retrieved from Vault at sync time.

The External Secrets Operator approach is the most flexible for teams already using a centralized secrets backend. The Application in ArgoCD deploys ExternalSecret objects, which the ESO controller resolves at runtime without ever touching Git.

Production Best Practices

  • Run ArgoCD in HA mode. Use manifests/ha/install.yaml with 3 replicas of the API server and multiple application controller shards for large clusters (100+ applications).
  • Pin image versions. Never use latest for the ArgoCD image itself. Pin to a specific version and upgrade deliberately.
  • Use the App of Apps pattern for bootstrapping. A single root Application deploys all other Applications. This makes cluster bootstrapping idempotent and reproducible.
  • Separate ArgoCD config from application config. Store ArgoCD Application manifests in a dedicated gitops repository, separate from application source code.
  • Enable resource tracking via annotations. Use application.resourceTrackingMethod: annotation in argocd-cm instead of the default label-based tracking, which can conflict with Helm’s own labels.
  • Set resource limits on ArgoCD controllers. Application controller CPU and memory scale with the number of resources tracked. Monitor and tune accordingly.
  • Restrict auto-sync in production. Consider requiring manual sync approval for production environments even when using GitOps — or at minimum require a PR approval gate before changes reach the target branch.

ArgoCD vs Flux

Flux v2 is the other major GitOps operator. Both are CNCF projects. The main differences in practice:

FeatureArgoCDFlux v2
UIBuilt-in web UINo official UI (use Weave GitOps)
Multi-clusterSingle control plane manages many clustersAgent per cluster, pull model
ApplicationSetsNativeKustomization + HelmRelease
Secret managementPlugin-basedSOPS native integration
Learning curveSteeper (more concepts)Lower (Kubernetes-native CRDs)
CNCF statusGraduatedGraduated

ArgoCD wins when you need the UI, multi-cluster management from a central plane, or have a large operations team that benefits from the visual application topology view. Flux wins when you want a simpler, purely Kubernetes-native approach with better SOPS integration for secret management.

FAQ

Can ArgoCD deploy to the cluster it runs in?

Yes. The https://kubernetes.default.svc destination refers to the local cluster. ArgoCD can manage both its own cluster and external clusters simultaneously.

Does ArgoCD support private Git repositories?

Yes. Configure repository credentials via argocd repo add with SSH keys, HTTPS username/password, or GitHub App credentials. Credentials are stored as Kubernetes secrets in the ArgoCD namespace.

How does ArgoCD handle CRD installation?

CRDs can be managed by ArgoCD, but there is a chicken-and-egg problem: if a CRD is not yet installed, ArgoCD cannot validate resources that use it. The recommended pattern is to put CRDs in wave 1 and dependent resources in wave 2, or to use a separate Application for CRDs.

What is the difference between an Application and an AppProject?

An Application is the unit of deployment — it maps a Git source to a cluster destination. An AppProject is a grouping and access control boundary — it restricts what sources and destinations an Application within the project can use. Every Application belongs to exactly one AppProject.

How do I roll back a deployment with ArgoCD?

The GitOps way: revert the commit in Git and let ArgoCD reconcile. ArgoCD also provides a UI-based rollback to any previous sync revision, but this is considered a temporary measure — the Git history should always be updated to match.

Getting Started

The fastest path from zero to a working ArgoCD setup on a local cluster:

# 1. Create a local cluster (kind or minikube)
kind create cluster --name argocd-demo

# 2. Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# 3. Wait for pods
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=120s

# 4. Get the initial admin password
argocd admin initial-password -n argocd

# 5. Port-forward and log in
kubectl port-forward svc/argocd-server -n argocd 8080:443 &
argocd login localhost:8080 --username admin --insecure

# 6. Deploy your first application
argocd app create guestbook 
  --repo https://github.com/argoproj/argocd-example-apps.git 
  --path guestbook 
  --dest-server https://kubernetes.default.svc 
  --dest-namespace guestbook 
  --sync-policy automated

From here, the natural next steps are integrating ArgoCD with your existing CI pipeline (CI builds and pushes the image, updates the image tag in Git, ArgoCD detects the change and syncs), configuring SSO via Dex, and setting up the App of Apps pattern for managing multiple applications declaratively.

For teams looking to go deeper on GitOps and ArgoCD in production, the Kubernetes architecture patterns guide covers how ArgoCD fits into a broader platform engineering stack alongside service mesh, policy enforcement, and observability tooling.

Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each

Developer inspecting a distroless container with magnifying glass

The container works fine in CI. It deploys successfully to staging. Then something goes wrong in production and you type the command you always type: kubectl exec -it my-pod -- /bin/bash. The response is immediate: OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory.

You try /bin/sh. Same error. You try ls. Same error. The container image is distroless — it ships only your application binary and its runtime dependencies, with no shell, no package manager, no debugging tools of any kind. This is intentional and correct from a security standpoint. It is also a significant operational challenge the first time you face it in production.

This article covers every practical technique for debugging distroless containers in Kubernetes: kubectl debug with ephemeral containers (the standard approach), pod copy strategy (for Kubernetes versions without ephemeral container support, or when you need to modify the running pod spec), debug image variants (the pragmatic developer shortcut), cdebug (a purpose-built tool that simplifies the process), and node-level debugging (the last resort with the most power). For each technique I will explain what it can and cannot do, what Kubernetes version or RBAC permissions it requires, and in which scenario — developer in local, platform engineer in staging, ops in production — it is the appropriate choice.

Why Distroless Breaks the Normal Debugging Workflow

Traditional container debugging assumes you can exec into the container and use shell tools: ps, netstat, strace, curl, a text editor. Distroless images remove all of this by design. The Google distroless project, Chainguard’s Wolfi-based images, and the broader minimal image ecosystem deliberately exclude everything that is not required to run the application. The result is a dramatically smaller attack surface: no shell means no RCE via shell injection, no package manager means no easy escalation path, fewer binaries means fewer CVEs in the image scan.

The tradeoff is operational: when something goes wrong, you cannot use the tools that the process itself is not allowed to run. A Java application in gcr.io/distroless/java17-debian12 has the JRE and nothing else. A Go binary compiled with CGO disabled and shipped in gcr.io/distroless/static-debian12 has literally only the binary and the necessary CA certificates and timezone data. There is no wget to download a debug binary, no apt to install one, no bash to run a script.

Kubernetes solves this at the platform level with ephemeral containers, added as stable in Kubernetes 1.25. The principle is that a debug container — which can have a full shell and any tools you want — can be injected into a running pod and share its process namespace, network namespace, and filesystem mounts without modifying the original container or restarting the pod.

Option 1: kubectl debug with Ephemeral Containers

Ephemeral containers are the canonical solution. Since Kubernetes 1.25 (stable), kubectl debug can inject a temporary container into a running pod. The container shares the target pod’s network namespace by default, and with --target it can also share the process namespace of a specific container, allowing you to inspect its running processes and open file descriptors.

The basic invocation is:

kubectl debug -it my-pod \
  --image=busybox:latest \
  --target=my-container

The --target flag is the critical piece. Without it, the ephemeral container gets its own process namespace. With it, it shares the process namespace of the specified container — meaning you can run ps aux and see the application’s processes, use ls -la /proc/<pid>/fd to inspect open file descriptors, and read the application’s environment via cat /proc/<pid>/environ.

For a more capable debug environment, replace busybox with a richer image:

kubectl debug -it my-pod \
  --image=nicolaka/netshoot \
  --target=my-container

nicolaka/netshoot includes tcpdump, curl, dig, nmap, ss, iperf3, and dozens of other network diagnostic tools, making it the standard choice for network debugging scenarios.

What You Can and Cannot Do

Ephemeral containers share the pod’s network namespace and, when --target is used, the process namespace. This gives you:

  • Full visibility into the application’s network traffic from inside the pod (tcpdump, ss, netstat)
  • Process inspection via /proc/<pid> — open files, memory maps, environment variables, CPU/memory usage
  • Access to the pod’s DNS resolution context — exactly the same /etc/resolv.conf the application sees
  • Ability to make outbound network calls from the same network namespace (testing service endpoints, DNS resolution)

What you do not get with ephemeral containers:

  • Access to the application container’s filesystem. The ephemeral container has its own root filesystem. You cannot cat /app/config.yaml from the application container’s filesystem unless you access it via /proc/<pid>/root/.
  • Ability to remove the container once added. Ephemeral containers are permanent until the pod is deleted. This is by design — the Kubernetes API does not allow removing them after creation.
  • Volume mount modifications via CLI. You cannot add volume mounts to an ephemeral container via kubectl debug (though the API spec supports it, the CLI does not expose this).
  • Resource limits. Ephemeral containers do not support resource requests and limits in the kubectl debug CLI, though this is evolving.

Accessing the Application Filesystem

The most common surprise for developers new to ephemeral containers is that they cannot directly browse the application container’s filesystem. The workaround is the /proc filesystem:

# Find the application's PID
ps aux

# Browse its filesystem via /proc
ls /proc/1/root/app/
cat /proc/1/root/etc/config.yaml

# Or set the root to the application's root
chroot /proc/1/root /bin/sh  # only if /bin/sh exists in the app image

The /proc/<pid>/root path is a symlink to the container’s root filesystem as seen from the process namespace. Because the ephemeral container shares the process namespace with --target, the application’s PID is typically 1, and /proc/1/root gives you full read access to its filesystem.

RBAC Requirements

Ephemeral containers require the pods/ephemeralcontainers subresource permission. This is separate from pods/exec, which controls kubectl exec. A common mistake is to grant pods/exec for debugging purposes without realizing that ephemeral containers require an additional grant:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ephemeral-debugger
rules:
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

In production environments, this permission should be tightly scoped: time-limited via RoleBinding rather than permanent ClusterRoleBinding, restricted to specific namespaces, and ideally gated behind an approval workflow. The debug container runs as root by default, which can create privilege escalation paths if the application container runs as a non-root user with shared process namespace — the debug container can attach to the application’s processes with higher privileges.

Option 2: kubectl debug –copy-to (Pod Copy Strategy)

When you need to modify the pod’s container spec — replace the image, change environment variables, add a sidecar with a shared filesystem — the --copy-to flag creates a full copy of the pod with your modifications applied:

kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=my-app:debug \
  --share-processes

This creates a new pod named my-pod-debug that is a copy of my-pod but with the container image replaced by my-app:debug. If my-app:debug is your application image built with debug tooling included (or a debug variant from your registry), this lets you interact with the exact same binary in the exact same configuration as the original pod.

A more common use of --copy-to is to attach a debug container alongside the existing application container while keeping the original image unchanged:

kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  --container=debugger

This creates the copy-pod with both the original containers and a new debugger container sharing the process namespace. Unlike ephemeral containers, this approach supports volume mounts and resource limits, and the debug pod can be deleted cleanly when you are done.

Limitations of the Copy Strategy

The pod copy approach has a critical limitation: it is not debugging the original pod. It creates a new pod that may behave differently because:

  • It does not share the original pod’s in-memory state — if the issue is a goroutine leak or heap corruption that has been accumulating for hours, the fresh copy will not exhibit it immediately
  • It creates a new Pod UID, which means any admission webhooks, network policies, or pod-level security contexts that depend on pod identity may apply differently
  • If the original pod is crashing (CrashLoopBackOff), the copy will also crash — this technique does not help for crash debugging unless you also change the entrypoint

For crash debugging specifically, combine --copy-to with a modified entrypoint to keep the container alive:

kubectl debug my-crashing-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  -- sleep 3600

Option 3: Debug Image Variants

The most pragmatic approach — and the one most appropriate for developer workflows — is to maintain a debug variant of your application image that includes shell tooling. Both the Google distroless project and Chainguard provide this pattern officially.

Google distroless images have a :debug tag that adds BusyBox to the image:

# Production image
FROM gcr.io/distroless/java17-debian12

# Debug variant — identical but with BusyBox shell
FROM gcr.io/distroless/java17-debian12:debug

Chainguard images follow a similar convention with :latest-dev variants that include apk, a shell, and common utilities:

# Production (zero shell, minimal footprint)
FROM cgr.dev/chainguard/go:latest

# Development/debug variant
FROM cgr.dev/chainguard/go:latest-dev

If you build your own base images, the recommended approach is to use multi-stage builds and maintain separate build targets:

FROM golang:1.22 AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp .

# Production: static distroless image
FROM gcr.io/distroless/static-debian12 AS production
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

# Debug variant: same binary, with shell tools
FROM gcr.io/distroless/static-debian12:debug AS debug
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

In your CI/CD pipeline, build both targets and push my-app:${VERSION} (production) and my-app:${VERSION}-debug (debug variant) to your registry. The debug image is never deployed to production by default, but it exists and is ready to be used with kubectl debug --copy-to when needed.

Security Considerations for Debug Variants

Debug image variants defeat much of the security benefit of distroless if they are used in production, even temporarily. Track usage carefully: log when debug images are deployed, require explicit approval, and ensure they are removed after the debugging session. In regulated environments, consider whether deploying a debug variant to production namespaces is permitted by your security policy — in many cases it is not, and you must use ephemeral containers (which add a debug process to the pod without modifying the application image) instead.

Option 4: cdebug

cdebug is an open-source CLI tool that simplifies distroless debugging by wrapping kubectl debug with more ergonomic defaults and additional capabilities. Its primary value is in making ephemeral container debugging feel like a native shell experience:

# Install
brew install cdebug
# or: go install github.com/iximiuz/cdebug@latest

# Debug a running pod
cdebug exec -it my-pod

# Specify a namespace and container
cdebug exec -it -n production my-pod -c my-container

# Use a specific debug image
cdebug exec -it my-pod --image=nicolaka/netshoot

What cdebug adds over raw kubectl debug:

  • Automatic filesystem chroot. cdebug exec automatically sets the filesystem root of the debug container to the target container’s filesystem, so you browse / and see the application’s files — not the debug image’s files. This addresses the most common friction point with kubectl debug.
  • Docker integration. cdebug exec works identically for Docker containers (cdebug exec -it <container-id>), making it the same muscle memory for local and cluster debugging.
  • No RBAC complications for Docker-based local development — useful for developer workflows before the code reaches Kubernetes.

The tradeoff: cdebug is a third-party dependency and requires installation. In environments with strict tooling policies (regulated industries, air-gapped clusters), it may not be an option. In those cases, the raw kubectl debug workflow with /proc/1/root filesystem navigation is the baseline.

Option 5: Node-Level Debugging

When everything else fails — the pod is in CrashLoopBackOff too fast to attach to, the issue is a kernel-level problem, or you need tools like strace that require elevated privileges — node-level debugging gives you direct access to the container’s processes from the host node.

kubectl debug node/ creates a privileged pod on the target node that mounts the node’s root filesystem under /host:

kubectl debug node/my-node-name \
  -it \
  --image=nicolaka/netshoot

From this privileged pod, you can use nsenter to enter the namespaces of any container running on the node:

# Find the container's PID on the node
# (from within the node debug pod)
crictl ps | grep my-container
crictl inspect <container-id> | grep pid

# Enter the container's namespaces
nsenter -t <pid> -m -u -i -n -p -- /bin/sh

# Or just the network namespace (for network debugging)
nsenter -t <pid> -n -- ip a

The nsenter approach lets you run tools from the node’s or debug container’s toolset while operating in the namespaces of the target container. This is how you run strace against a distroless process: strace is not in the application container, but you can run it from the node level while targeting the application’s PID.

# Trace all syscalls from the application process
nsenter -t <pid> -- strace -p <pid> -f -e trace=network

RBAC and Security for Node Debugging

Node-level debugging requires nodes/proxy and the ability to create privileged pods, which in most production clusters is restricted to cluster administrators. The debug pod runs with hostPID: true and hostNetwork: true, giving it visibility into all processes and network traffic on the node — not just the target container. This is significant: every process running on the node, including those in other tenants’ namespaces, is visible.

This technique should be treated as a break-glass procedure: log the access, require dual approval in production environments, and clean up immediately after the debugging session with kubectl delete pod --selector=app=node-debugger.

Choosing the Right Approach: Access Profile and Environment Matrix

The technique you should use depends on two axes: who you are (developer, platform engineer, ops/SRE) and where the issue is (local development, staging, production). The requirements and constraints differ significantly across these combinations.

Developer — Local or Development Cluster

Goal: Reproduce and understand a bug, inspect configuration, verify network connectivity to services.
Constraints: None material — full cluster admin on local or personal dev namespace.
Recommended approach: Debug image variants or cdebug.

In local development (Minikube, Kind, Docker Desktop), the fastest path is to build the debug variant of your image and deploy it directly. If you are working with another team’s service, cdebug exec gives you a shell in the container with automatic filesystem root without any special RBAC. The goal is speed and iteration — reserve the more structured approaches for higher environments.

Developer — Staging Cluster

Goal: Debug integration issues, inspect live configuration, verify environment-specific behavior.
Constraints: Shared cluster — cannot deploy arbitrary workloads to other teams’ namespaces, but has pods/ephemeralcontainers in own namespace.
Recommended approach: kubectl debug with ephemeral containers (--target), scoped to own namespace.

Staging is where ephemeral containers earn their keep. You can attach to a running pod without restarting it, without modifying the deployment spec, and without affecting other users of the same cluster. Grant developers pods/ephemeralcontainers in their team’s namespaces and they can self-service debug without needing ops involvement.

Platform Engineer / SRE — Production

Goal: Diagnose a live production incident. The pod is behaving unexpectedly — high latency, memory growth, unexpected connections, incorrect responses.
Constraints: Changes to running pods are high-risk. Any debug image deployment must be gated. The issue is live and affecting users.
Recommended approach: kubectl debug with ephemeral containers (ephemeral containers do not restart the pod, do not modify the deployment, and are auditable via API audit logs).

The key production requirements are auditability and minimal blast radius. Ephemeral containers satisfy both: they are recorded in the Kubernetes API audit log (who attached, when, to which pod), they do not modify the running application container, and they are limited to the pod’s own network and process namespaces. Document the debug session in your incident ticket: pod name, time, what was observed, who ran the debug container.

The --copy-to strategy is generally inappropriate for production incident response: it creates a new pod that may or may not exhibit the issue, it adds load to the cluster during an incident, and if it is attached to the same services (databases, downstream APIs), it produces additional traffic that complicates forensics.

Platform Engineer — Production, Node-Level Issue

Goal: Diagnose a kernel-level issue, a container runtime problem, a networking issue that spans multiple pods, or a situation where the pod is crashing too fast to attach to.
Constraints: Maximum privilege required. High operational risk.
Recommended approach: Node-level debug pod with nsenter. Treat as break-glass.

For this scenario, create a dedicated RBAC role that grants nodes/proxy access and the ability to create pods with hostPID: true in a dedicated debug namespace. Bind it only to specific users, require a separate authentication step (e.g., kubectl auth can-i check against a time-limited binding), and log all access. This level of access should generate a PagerDuty-style alert so that the security team knows a privileged debug session is active in production.

Common Errors and Solutions

Error: “ephemeral containers are disabled for this cluster”

Ephemeral containers require Kubernetes 1.16+ (alpha, behind feature gate) and are stable from 1.25. If you are on 1.16–1.22, you need to enable the EphemeralContainers feature gate on the API server and kubelet. From 1.23 it was beta and enabled by default. From 1.25 it is stable and always on. On managed Kubernetes services (EKS, GKE, AKS), check the cluster version — versions older than 1.25 may still have it disabled depending on your configuration.

Error: “cannot update ephemeralcontainers” (RBAC)

You have pods/exec but not pods/ephemeralcontainers. Add the grant shown in the RBAC section above. Note that pods/exec and pods/ephemeralcontainers are separate subresources — having one does not imply the other.

Error: “container not found” with –target

The container name in --target must match exactly the container name as defined in the Pod spec — not the image name. Check with kubectl get pod my-pod -o jsonpath='{.spec.containers[*].name}' to get the exact container names.

Error: Can see processes but cannot read /proc/1/root

The application container runs as a non-root user (e.g., UID 1000) and the ephemeral container runs as root. The application’s filesystem may have files owned by UID 1000 that are not readable by other UIDs depending on permissions. The /proc/<pid>/root path itself requires CAP_SYS_PTRACE capability. If your cluster’s PodSecurityStandards (PSS) are set to restricted, the debug container may not have this capability. Use the Baseline PSS profile for debug namespaces or explicitly add SYS_PTRACE to the ephemeral container’s securityContext.

Error: tcpdump shows no traffic

When using nicolaka/netshoot for network debugging, ensure the ephemeral container is created without --target if your goal is to capture all traffic on the pod’s network interface (not just the specific container’s process). With --target, you share the process namespace but the network namespace is shared at the pod level regardless. Run tcpdump -i any to capture on all interfaces including loopback, which is where inter-container traffic within a pod travels.

Decision Framework

Use this as a starting point to select the right technique for your situation:

ScenarioTechniqueRequirement
Active production incident, pod runningkubectl debug + ephemeral containerpods/ephemeralcontainers RBAC, k8s 1.25+
Pod crashing too fast to attachkubectl debug –copy-to + modified entrypointAbility to create pods in namespace
Developer debugging in dev/stagingcdebug exec or kubectl debugpods/ephemeralcontainers or pod create
Need full filesystem accesskubectl debug –copy-to + debug image variantDebug image in registry, pod create
Need strace or kernel tracingNode-level debug with nsenternodes/proxy, cluster admin equivalent
Network packet capturekubectl debug + nicolaka/netshootpods/ephemeralcontainers
Local Docker debuggingcdebug exec <container-id>Docker socket access
CI-reproducible debug environmentDebug image variant in separate build targetSeparate image tag in registry

Production RBAC Design

A clean RBAC design for production distroless debugging separates three roles with different privilege levels:

# Tier 1: Developer self-service in team namespaces
# Allows attaching ephemeral containers, no node access
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: distroless-debugger
  namespace: team-namespace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
---
# Tier 2: SRE production incident access
# Ephemeral containers across all namespaces
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sre-distroless-debugger
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
---
# Tier 3: Break-glass node access
# Only for platform team, time-limited binding recommended
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-debugger
rules:
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create", "get", "list", "delete"]
  # Restrict to debug namespace via RoleBinding, not ClusterRoleBinding

Bind Tier 1 permanently to your developers. Bind Tier 2 to SREs permanently but with audit alerts on use. Bind Tier 3 only on-demand (via a Kubernetes operator that creates time-limited RoleBindings) and never as a permanent ClusterRoleBinding.

Summary

Distroless containers are the correct choice for production workloads. They reduce attack surface, eliminate unnecessary CVEs, and force a cleaner separation between application and tooling. The operational cost is that your traditional debugging workflow — exec into the container, run some commands — no longer works by default.

Kubernetes provides a clean answer with ephemeral containers and kubectl debug: inject a debug container with whatever tools you need into the running pod, sharing its network and process namespaces, without restarting or modifying the application. For scenarios where ephemeral containers are insufficient — filesystem access, crash debugging, kernel-level investigation — the copy strategy and node-level debug fill the remaining gaps.

The key to making this work at scale is not the technique itself but the access model: developers get self-service ephemeral container access in their own namespaces, SREs get cluster-wide ephemeral container access for production incidents, and node-level access is a break-glass procedure with audit trail and time limits. With that model in place, distroless becomes an operational non-issue rather than an obstacle.

Building a Kubernetes Migration Framework: Lessons from Ingress-NGINX

Building a Kubernetes Migration Framework: Lessons from Ingress-NGINX

The recent announcement regarding the deprecation of the Ingress-NGINX controller sent a ripple through the Kubernetes community. For many organizations, it’s the first major deprecation of a foundational, widely-adopted ecosystem component. While the immediate reaction is often tactical—”What do we replace it with?”—the more valuable long-term question is strategic: “How do we systematically manage this and future migrations?”

This event isn’t an anomaly; it’s a precedent. As Kubernetes matures, core add-ons, APIs, and patterns will evolve or sunset. Platform engineering teams need a repeatable, low-risk framework for navigating these changes. Drawing from the Ingress-NGINX transition and established deployment management principles, we can abstract a robust Kubernetes Migration Framework applicable to any major component, from service meshes to CSI drivers.

Why Ad-Hoc Migrations Fail in Production

Attempting a “big bang” replacement or a series of manual, one-off changes is a recipe for extended downtime, configuration drift, and undetected regression. Production Kubernetes environments are complex systems with deep dependencies:

  • Interdependent Workloads: Multiple applications often share the same ingress controller, relying on specific annotations, custom snippets, or behavioral quirks.
  • Automation and GitOps Dependencies: Helm charts, Kustomize overlays, and ArgoCD/Flux manifests are tightly coupled to the existing component’s API and schema.
  • Observability and Security Integration: Monitoring dashboards, logging parsers, and security policies are tuned for the current implementation.
  • Knowledge Silos: Tribal knowledge about workarounds and specific configurations isn’t documented.

A structured framework mitigates these risks by enforcing discipline, creating clear validation gates, and ensuring the capability to roll back at any point.

The Four-Phase Kubernetes Migration Framework

This framework decomposes the migration into four distinct phases: Assessment, Parallel Run, Cutover, and Decommission. Each phase has defined inputs, activities, and exit criteria.

Phase 1: Deep Assessment & Dependency Mapping

Before writing a single line of new configuration, understand the full scope. The goal is to move from “we use Ingress-NGINX” to a precise inventory of how it’s used.

  • Inventory All Ingress Resources: Use kubectl get ingress --all-namespaces as a starting point, but go deeper.
  • Analyze Annotation Usage: Script an analysis to catalog every annotation in use (e.g., nginx.ingress.kubernetes.io/rewrite-target, nginx.ingress.kubernetes.io/configuration-snippet). This reveals functional dependencies.
  • Map to Backend Services: For each Ingress, identify the backend Services and Namespaces. This highlights critical applications and potential blast radius.
  • Review Customizations: Document any custom ConfigMaps for main NGINX configuration, custom template patches, or modifications to the controller deployment itself.
  • Evaluate Alternatives: Based on the inventory, evaluate candidate replacements (e.g., Gateway API with a compatible implementation, another Ingress controller like Emissary-ingress or Traefik). The Google Cloud migration framework provides a useful decision tree for ingress-specific migrations.

The output of this phase is a migration manifesto: a concrete list of what needs to be converted, grouped by complexity and criticality.

Phase 2: Phased Rollout & Parallel Run

This is the core of a low-risk migration. Instead of replacing, you run the new and old systems in parallel, shifting traffic gradually. For ingress, this often means installing the new controller alongside the old one.

  • Dual Installation: Deploy the new ingress controller in the same cluster, configured with a distinct ingress class (e.g., ingressClassName: gateway vs. nginx).
  • Create Canary Ingress Resources: For a low-risk application, create a parallel Ingress or Gateway resource pointing to the new controller. Use techniques like managed deployments with canary patterns to control exposure.
    # Example: A new Gateway API HTTPRoute for a canary service
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
    name: app-canary
    spec:
    parentRefs:
    - name: company-gateway
    rules:
    - backendRefs:
    - name: app-service
    port: 8080
    weight: 10 # Start with 10% of traffic

  • Validate Equivalency: Use traffic mirroring (if supported) or direct synthetic testing against both ingress paths. Compare logs, response headers, latency, and error rates.
  • Iterate and Expand: Gradually increase traffic weight or add more applications to the new stack, group by group, based on the assessment from Phase 1.

This phase relies heavily on your observability stack. Dashboards comparing error rates, latency (p50, p99), and throughput between the old and new paths are essential.

Phase 3: Validation & Automated Cutover

The cutover is not a manual event. It’s the final step in a validation process.

  • Define Validation Tests: Create a suite of tests that must pass before full cutover. This includes:
    • Smoke tests for all critical user journeys.
    • Load tests to verify performance under expected traffic patterns.
    • Security scan validation (e.g., no unintended ports open).
    • Compliance checks (e.g., specific headers are present).
  • Automate the Switch: For each application, the cutover is ultimately a change in its Ingress or Gateway resource. This should be done via your GitOps pipeline. Update the source manifests (e.g., change the ingressClassName), merge, and let automation apply it. This ensures the state is declarative and recorded.
  • Maintain Rollback Capacity: The old system must remain operational and routable (with reduced capacity) during this phase. The GitOps rollback is simply reverting the manifest change.

Phase 4: Observability & Decommission

Once all traffic is successfully migrated and validated over a sustained period (e.g., 72 hours), you can decommission the old component.

  • Monitor Aggressively: Keep a close watch on all key metrics for at least one full business cycle (a week).
  • Remove Old Resources: Delete the old controller’s Deployment, Service, ConfigMaps, and CRDs (if no longer needed).
  • Clean Up Auxiliary Artifacts: Remove old RBAC bindings, service accounts, and any custom monitoring alerts or dashboards specific to the old component.
  • Document Lessons Learned: Update runbooks and architecture diagrams. Note any surprises, gaps in the process, or validation tests that were particularly valuable.

Key Principles for a Resilient Framework

Beyond the phases, these principles should guide your framework’s design:

  • Always Maintain Rollback Capability: Every step should be reversible with minimal disruption. This is a core tenet of managing Kubernetes deployments.
  • Leverage GitOps for State Management: All desired state changes (Ingress resources, controller deployments) must flow through version-controlled manifests. This provides an audit trail, consistency, and the simplest rollback mechanism (git revert).
  • Validate with Production Traffic Patterns: Synthetic tests are insufficient. Use canary weights and traffic mirroring to validate with real user traffic in a controlled manner.
  • Communicate Transparently: Platform teams should maintain a clear migration status page for internal stakeholders, showing which applications have been migrated, which are in progress, and the overall timeline.

Conclusion: Building a Migration-Capable Platform

The deprecation of Ingress-NGINX is a wake-up call. The next major change is a matter of “when,” not “if.” By investing in a structured migration framework now, platform teams transform a potential crisis into a manageable, repeatable operational procedure.

This framework—Assess, Run in Parallel, Validate, and Decommission—abstracts the specific lessons from the ingress migration into a generic pattern. It can be applied to migrating from PodSecurityPolicies to Pod Security Standards, from a deprecated CSI driver, or from one service mesh to another. The tools (GitOps, canary deployments, observability) are already in your stack. The value is in stitching them together into a disciplined process that ensures platform evolution doesn’t compromise platform stability.

Start by documenting this framework as a runbook template. Then, apply it to your next significant component update, even a minor one, to refine the process. When the next major deprecation announcement lands in your inbox, you’ll be ready.