Talos Linux vs Bottlerocket vs Flatcar Container Linux: choosing an immutable OS for Kubernetes nodes (2026)

Talos Linux vs Bottlerocket vs Flatcar Container Linux: choosing an immutable OS for Kubernetes nodes (2026)

Kubernetes node operating systems have become a real design choice again. For years, most teams ran Ubuntu, Debian, RHEL, or Amazon Linux on workers, then layered kubelet, containerd, hardening, patching, observability agents, GPU drivers, and configuration management on top. That works, but it leaves a large mutable host underneath a scheduler that treats nodes as replaceable capacity.

Immutable container-focused operating systems attack the problem differently. They reduce the host to a small, versioned artifact, make updates atomic, and move configuration into declarative APIs or boot-time documents. The goal is fewer moving parts, less drift, smaller attack surface, predictable upgrades, and replaceable nodes.

In 2026, the three names that come up most often are Talos Linux, Bottlerocket, and Flatcar Container Linux. They overlap, but they are not interchangeable:

  • Talos Linux is the most Kubernetes-specific and the most radical: no SSH, no shell, no package manager, API-only operations.
  • Bottlerocket is the AWS-native container host: strong fit for EKS and ECS, image variants, API settings, admin containers, and AWS support.
  • Flatcar Container Linux is the evolutionary successor to the CoreOS Container Linux model: read-only system partition, Ignition/Container Linux Config, systemd, SSH, automatic updates, and broad platform coverage.

For a step-by-step Talos implementation guide, read the deep dive: Talos Linux: The Immutable, API-Driven OS for Kubernetes. This article is a decision guide: what changes operationally, where each OS fits, and what trade-offs senior platform teams should care about.

The decision in one paragraph

Choose Bottlerocket when your platform is AWS-based and you want a supported node OS that integrates with EKS, EC2, ECS, AMIs, and AWS support plans. Choose Talos when you want a Kubernetes-first OS across bare metal, edge, private cloud, or multi-cloud, and can replace SSH-era habits with an API-driven model. Choose Flatcar when you want immutable infrastructure and automatic updates, but still need systemd, SSH, Ignition-style provisioning, and a smoother migration from Ubuntu or CoreOS-like patterns.

What Talos Linux is

Talos Linux is a Kubernetes-optimized Linux distribution from Sidero Labs. Its design is deliberately narrow: it exists to run Kubernetes. It is not a general-purpose server distribution with a hardened profile. The host has no SSH daemon, no interactive shell, and no package manager. Configuration and day-2 operations go through talosctl and an authenticated gRPC API using mutual TLS.

That changes the security model. On a traditional node, operational access usually means SSH, sudo, shell history, mutable filesystems, ad hoc commands, and sometimes a long tail of packages that were installed to debug one incident and never removed. Talos removes that path. You inspect resources, logs, services, disks, and Kubernetes component state through the Talos API. You change machine configuration by applying declarative config patches. You upgrade by asking the node to move to a new installer image.

The upside is a very small node surface. Sidero states that Talos ships with fewer than 50 OS binaries and no package manager; current releases also publish SBOM, VEX, and cryptographic signatures. Talos upgrades use an A/B image scheme so the previous kernel and OS image remain available for rollback. As of this review, the latest GitHub release is Talos v1.13.5, published on June 22, 2026.

The cost is operational discipline. If your incident response muscle memory is ssh, strace, edit a file, restart a daemon, Talos forces a redesign around API resources, Kubernetes-level observability, ephemeral debug containers where appropriate, and reproducible configuration.

What Bottlerocket is

Bottlerocket is an open-source Linux-based operating system purpose-built by Amazon Web Services for running containers. AWS provides Bottlerocket as no-cost AMIs for EC2 and documents it as a container host for EKS, ECS, VMware, and bare metal. The product center of gravity is clearly AWS.

Bottlerocket does not use a general-purpose package manager. Instead, it ships as variants: predefined images tailored to platform, architecture, orchestrator, orchestrator version, and sometimes flavor. Kubernetes variants are tied to Kubernetes minor versions, and Bottlerocket also has NVIDIA and FIPS flavors. The Bottlerocket OS version is separate from the Kubernetes variant version. As of this review, the current docs are under the 1.62.x branch and the latest GitHub release is v1.62.1, published on June 22, 2026.

Configuration is modeled and exposed through an API. Updates are image-based and use partition flips rather than package mutation. For Kubernetes clusters, the Bottlerocket Update Operator, or Brupop, coordinates in-place node updates. It runs a controller, node agents, and an API server, drains nodes through the eviction API, respects PodDisruptionBudgets, and rolls updates in waves.

Bottlerocket is less austere than Talos. It supports host and admin container concepts for controlled access, which helps with break-glass paths, SSM-style workflows, and AWS support expectations. The trade-off is portability: it can run outside EC2, but its strongest ecosystem, release consumption model, and support story are AWS-shaped.

What Flatcar Container Linux is

Flatcar Container Linux is a community Linux distribution for container workloads. It started as a fork of CoreOS Container Linux and now lives in the Cloud Native Computing Foundation. CNCF lists Flatcar as an Incubating project, accepted on August 2, 2024.

Flatcar is immutable in the classic Container Linux sense. The system partition is read-only, the OS is delivered as an image, updates are automatic and atomic, and the node is provisioned declaratively at boot. Unlike Talos, Flatcar keeps a familiar Linux model: systemd, SSH, files, units, and host-level customization. The current recommended provisioning path is Container Linux Config transpiled to Ignition. The older cloud-config model remains useful historically, but the docs recommend Container Linux Config for new provisioning.

Flatcar’s platform coverage is broad, including major clouds, OpenStack, Proxmox, VMware, KubeVirt, libvirt, QEMU, and VirtualBox. The release page currently shows Stable 4593.2.3, Beta 4694.1.0, Alpha 4722.0.0, and LTS 4081.3.8. The LTS model is useful for teams that want fewer feature changes.

Flatcar’s day-2 profile is closest to traditional Linux, which is both its strength and its risk. You can carry forward systemd-based automation, SSH debugging, and existing tools. You can also carry forward drift if you treat it like a mutable server.

Comparative table

AreaTalos LinuxBottlerocketFlatcar Container Linux
Core philosophyKubernetes-only, API-managed, no shellAWS-built container host with variantsContainer Linux successor with systemd and Ignition
Best fitBare metal, edge, private cloud, multi-cloud, high-security clustersEKS/ECS and AWS-heavy fleetsUbuntu/CoreOS-style migrations that still need Linux primitives
Management modeltalosctl over mTLS gRPC; declarative machine configAPI settings, variants, AMIs, BrupopContainer Linux Config/Ignition, systemd, SSH, channels, Nebraska
SSH and shellNo SSH, no shellNot a normal SSH-first OS; controlled access through host/admin containersSSH and normal Linux debugging patterns are available
Package managerNoneNone; use variants insteadNo traditional package workflow for the base OS; extend through containers, Ignition, systemd, and systemd-sysext
UpdatesAPI-triggered upgrades, A/B image scheme, rollback pathPartition-flip updates; Brupop for Kubernetes nodesAutomatic atomic updates via channels and reboot strategies
Kubernetes couplingVery high; control plane and workers are first-classHigh for Kubernetes variants; also ECSMedium; Kubernetes lifecycle is assembled above the OS
Attack surfaceSmallest and most opinionated; no SSH, no shell, no package managerSmall, container-focused, no package manager, AWS security postureMinimal container OS, read-only system partition, but broader Linux surface than Talos
Platform supportBroad: bare metal, VMs, cloud, edgeStrongest on AWS; also VMware and bare metal variantsBroad cloud and virtualization support
ExtensionsSystem extensions, Image Factory, extension services, release-bound GPU supportVariants and flavors such as NVIDIA and FIPSsystemd-sysext, containers, systemd units, sysext-bakery
Governance/companyOpen-source Talos by Sidero Labs; Sidero is a CNCF Silver MemberOpen-source project built by AWS; AWS supports AWS-provided buildsCNCF Incubating project since August 2024
Learning curveHighest if the team expects SSHModerate for AWS teams; lower inside EKS practicesLowest for Linux operations teams

Day-2 operations: upgrades, debugging, and extensions

The day-2 question matters more than installation. You will install the OS once per node lifecycle, but upgrade, debug, replace, patch, and explain incidents for years.

Talos is the most coherent if you want every node operation to be declarative and auditable. Upgrades are API calls against an installer image. Kubernetes and OS upgrades are explicit operations, not package side effects. Diagnostics come from talosctl: service status, logs, disks, network state, component health, and resource APIs. Runbooks need Talos resources, not shell commands.

Extensions in Talos are powerful but constrained. System extensions are included in generated boot media or installer images, commonly via Image Factory. Extension services can run early privileged services where static pods or DaemonSets are not enough. Hardware-specific work, such as NVIDIA support, is possible but release-bound.

Bottlerocket day-2 operations fit AWS lifecycle thinking. In EKS, Brupop handles in-place updates for existing Bottlerocket nodes. It drains nodes, uses the eviction API, respects PDBs, and limits rollout blast radius. New nodes still depend on the AMI or image you launch, so keep launch templates current.

Debugging Bottlerocket is less disruptive than Talos for teams that want controlled host access, because the OS supports admin and host containers. Treat that as break-glass access, not routine configuration.

Flatcar day-2 operations are the most familiar. Automatic updates arrive through channels. Production clusters usually use Stable; a subset can run Beta to catch issues earlier. Reboot behavior is controlled through documented strategies, and organizations can run their own Nebraska-style update infrastructure. Flatcar also has an LTS channel.

For extensions, Flatcar has become more modular through systemd-sysext: overlays on read-only /usr, prebuilt sysext-bakery images, Kubernetes binaries, CRI-O, K3s, Wasmtime, and cloud vendor tooling.

When to choose Talos

Choose Talos when Kubernetes is the product boundary of the node. If nothing should run on the host except Kubernetes machinery, Talos gives you the cleanest model. It is compelling for bare metal, edge clusters, private cloud, Proxmox/vSphere, and multi-cloud fleets where you want one operational abstraction.

Talos is also the strongest choice when security teams want a hard answer to SSH access, mutable root filesystems, package drift, and undocumented host changes. Pair it with Kubernetes security best practices and the node layer becomes easier to audit.

Do not choose Talos casually if your team depends on node-level manual debugging, package-installed host agents, or mixed workloads outside Kubernetes. Talos rewards teams that invest in runbooks, observability, automated replacement, and reproducible node configuration.

When to choose Bottlerocket

Choose Bottlerocket when most of your Kubernetes fleet runs on AWS, especially EKS, and your organization values AWS-native support paths. Bottlerocket aligns with EC2 images, EKS worker patterns, AWS-provided builds, and AWS support plans.

Bottlerocket also fits teams that want immutable nodes but cannot go all the way to Talos’s no-shell model. Admin and host containers provide controlled access, variants reduce ambiguity, and Brupop coordinates rollout.

The caveat is portability. Bottlerocket can run in non-AWS contexts, but the further you move from AWS, the more you should compare it against Talos and Flatcar on ecosystem, tooling, and support.

When to choose Flatcar

Choose Flatcar when you want a lower-risk transition from traditional Linux nodes to immutable infrastructure. It is strong for teams that already understand systemd, Ignition, CoreOS-style provisioning, SSH, update channels, and cloud images. It gives you a read-only base OS, automatic atomic updates, broad platform availability, and CNCF governance without rewriting every host-level assumption.

Flatcar is also useful when Kubernetes is not the only container workload, or when Kubernetes, standalone container hosts, and custom systemd-managed components coexist. It is less opinionated than Talos and less AWS-centered than Bottlerocket.

The risk is familiarity. If engineers SSH into nodes, add one-off files, and treat systemd units as snowflake configuration, the immutable base loses value. Manage Flatcar with boot-time config, versioned artifacts, controlled update channels, and replaceable nodes.

FAQ

Is Talos more secure than Bottlerocket or Flatcar?

It depends on what you mean by secure. Talos has the smallest operational attack surface because it removes SSH, shell access, and package management entirely. Bottlerocket and Flatcar also reduce attack surface compared with general-purpose distributions, but preserve different forms of host access and customization.

Can Bottlerocket run outside AWS?

Yes, but AWS remains its center of gravity. Official materials describe variants for Kubernetes worker nodes in EC2, VMware, and bare metal, and ECS variants for EC2. For non-AWS-first environments, compare platform support, image lifecycle, and support expectations against Talos and Flatcar.

Is Flatcar just old CoreOS?

No. Flatcar started as a fork of CoreOS Container Linux, but it is now an active CNCF Incubating project with its own release streams, platform support, update mechanisms, and systemd-sysext-based modularity.

Which one is easiest to migrate to from Ubuntu worker nodes?

Flatcar is usually easiest because it preserves systemd, SSH, and a recognizable Linux host model while adding immutable OS updates. Bottlerocket can be straightforward inside EKS. Talos is often the biggest shift because it changes how administrators access and debug nodes.

Final recommendation

For AWS-heavy EKS fleets, start with Bottlerocket unless portability or a no-shell requirement points elsewhere. For bare metal, edge, private cloud, and multi-cloud Kubernetes, put Talos at the top of the shortlist and validate debugging and extension workflows early. For teams modernizing from Ubuntu or legacy CoreOS-style infrastructure, Flatcar is the pragmatic bridge.

The bigger decision is whether your platform treats nodes as replaceable, versioned infrastructure. Once that is true, all three can work well. The best choice is the one whose failure modes match your team’s reality.

Sources

  • https://www.siderolabs.com/talos-linux
  • https://docs.siderolabs.com/talos/v1.13/overview/what-is-talos
  • https://docs.siderolabs.com/talos/v1.13/configure-your-talos-cluster/lifecycle-management/upgrading-talos
  • https://docs.siderolabs.com/talos/v1.13/build-and-extend-talos/custom-images-and-development/system-extensions
  • https://docs.siderolabs.com/talos/v1.13/build-and-extend-talos/custom-images-and-development/extension-services
  • https://docs.siderolabs.com/talos/v1.13/learn-more/image-factory
  • https://github.com/siderolabs/talos/releases
  • https://aws.amazon.com/bottlerocket/
  • https://aws.amazon.com/bottlerocket/faqs/
  • https://bottlerocket.dev/en/os/1.62.x/concepts/variants/
  • https://github.com/bottlerocket-os/bottlerocket
  • https://github.com/bottlerocket-os/bottlerocket/releases
  • https://github.com/bottlerocket-os/bottlerocket-update-operator
  • https://www.cncf.io/projects/flatcar-container-linux/
  • https://www.flatcar.org/
  • https://www.flatcar.org/releases
  • https://www.flatcar.org/docs/latest/provisioning/cl-config/from-cloud-config/
  • https://www.flatcar.org/docs/latest/setup/releases/update-strategies/
  • https://www.flatcar.org/blog/2024/04/os-innovation-with-systemd-sysext/

Stop Implementing Authentication Inside Containers on Kubernetes

Stop Implementing Authentication Inside Containers on Kubernetes

You have ten microservices running in Kubernetes. Each one validates JWTs, checks scopes, maintains sessions, and implements its own RBAC rules. One team uses jsonwebtoken v8, another uses a custom Go library, a third rolled their own HMAC check because “it was simple.” They all accept alg: none. Three accept RS256 and HS256 simultaneously.

This is not a security posture. This is a distributed security liability — and Kubernetes makes the problem worse, because the cluster creates an illusion of isolation that encourages teams to treat each Pod as a security boundary it was never designed to be.

The pattern of embedding authentication and authorization logic inside individual containers is one of the most pervasive anti-patterns in Kubernetes-based microservices. It feels like ownership and simplicity. It is, in practice, inconsistency at scale — and the blast radius of a single misconfiguration is your entire service portfolio.

Kubernetes provides the primitives to fix this at the infrastructure layer: Ingress controllers, the Gateway API, service mesh sidecars, admission webhooks, and workload identity via SPIFFE. None of these require a line of auth code inside your application containers.

This article explains why the anti-pattern exists, what’s wrong with it technically, and what the correct Kubernetes-native alternatives are — with concrete implementation guidance and references to the standards and incidents that validate the argument.


The Anti-Pattern: What It Looks Like

In-Application JWT Validation

Every service imports an auth library and validates tokens independently:

# Pattern seen in thousands of microservices
from jose import jwt

def authenticate(token: str):
    payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
    return payload["sub"]

Variations include:

  • Algorithm confusion: accepting both HS256 and RS256, or letting the token header drive verification behavior instead of pinning acceptable algorithms server-side. This is a distinct JWT implementation failure class, documented extensively by PortSwigger on JWT attacks and RFC 8725
  • alg: none bypass: libraries that accept unsigned tokens when alg is set to none. This is a documented attack vector in Auth0’s JWT security analysis
  • Missing exp / iss / aud validation: trusting a valid signature without checking whether the token is expired, for the right audience, or from the right issuer
  • Key confusion attacks: accepting a public RS256 key as an HS256 symmetric secret

Session State in Every Service

When services maintain session state directly, they duplicate logic that has no business being duplicated — cookie validation, refresh token flows, PKCE verification — and each implementation diverges over time.

RBAC Reimplemented Per Service

Authorization rules (“can this user access this resource?”) end up embedded in service logic, mixed with business logic, tested inconsistently, and impossible to audit across the portfolio.


Why This Is a Structural Problem

1. The Vulnerability Surface Scales With Your Service Count

Each new microservice is a new JWT validation surface. A single incorrect library configuration — an unvalidated alg, a missing aud check — is an authentication bypass affecting that entire service. With ten services, you have ten potential misconfigurations. With a hundred services, the probability that at least one is misconfigured approaches certainty.

The OWASP Kubernetes Security Cheat Sheet and OWASP Microservices Security Cheat Sheet both identify in-service auth as a primary attack surface in microservices environments. NIST SP 800-204 and its companion NIST SP 800-204A on DevSecOps make the same argument: security controls belong at infrastructure boundaries, not inside application code.

2. Maintenance Cost Is Multiplicative

When a JWT vulnerability is disclosed — and they are disclosed regularly — you update one library in one service. Then another. Then you discover service C is pinned to an old version because it has a transitive dependency conflict. Meanwhile the vulnerability is exploitable in production.

The CNCF Cloud Native Security Whitepaper frames this directly: security controls implemented redundantly across services create maintenance overhead that teams cannot sustain, leading to version drift and policy divergence.

3. Centralized Policy Is Impossible to Enforce

When policy is in code — even well-factored library code — it cannot be changed atomically across services. A policy update requires a coordinated deployment across every affected service. In practice, services deploy on different schedules, managed by different teams, with different testing cycles. The result is that at any given moment, some fraction of your services are running different authorization rules.

This is the core argument in Google’s BeyondCorp model and the Zero Trust Architecture guidance from NIST SP 800-207: authentication and authorization decisions should be made by a centralized, auditable policy enforcement point — not distributed across workloads.

4. Secrets Distribution Is a Problem You Don’t Need

If every service validates JWTs, every service needs the signing key (for symmetric algorithms) or the public key (for asymmetric). Distributing and rotating signing keys across a fleet of microservices is an operational burden with meaningful blast radius: a leaked symmetric key compromises every service holding it.

The CNCF SPIFFE/SPIRE project was built specifically to solve this class of problem: workload identity should be cryptographically attested, not rely on secrets distributed to application code.


The Real-World Incidents

The alg: none Class

In 2015, critical vulnerabilities in JWT libraries from Auth0 affecting Python, PHP, Node.js, Ruby, Java, and .NET allowed attackers to forge tokens by setting alg: none. The signature was not verified. The vulnerability was present in applications that had copied JWT validation code from tutorials or used unpatched libraries — exactly the pattern that in-service auth produces at scale.

Java’s Psychic Signatures (CVE-2022-21449)

CVE-2022-21449 affected ECDSA signature verification in Oracle Java SE and GraalVM, including java.security.Signature paths used by higher-level libraries. The bug allowed certain malformed ECDSA signatures to verify when they should not. JWT validation was in scope only when the deployment used an affected Java runtime and ECDSA-signed tokens, for example ES256. A gateway would help only if verification happened on a patched or unaffected runtime at the gateway instead of inside every Java service.

CVE-2023-2728 (Kubernetes Mountable Secrets Bypass)

CVE-2023-2728 was not an ImagePolicyWebhook outage behavior. It was a Kubernetes API server issue where users could use ephemeral containers to bypass the mountable secrets policy enforced by the ServiceAccount admission plugin. Clusters were affected only when the ServiceAccount admission plugin, the kubernetes.io/enforce-mountable-secrets annotation, and ephemeral containers were used together. The adjacent ImagePolicyWebhook issue was CVE-2023-2727, also involving ephemeral containers, but it is a separate CVE.

The Uber API Gateway Evolution

Uber’s engineering blog describes its API gateway as a centralized layer for routing, protocol conversion, rate limiting, load shedding, header propagation, security auditing, and user access blocking. That supports the architectural point here: high-volume platforms move cross-cutting controls into shared infrastructure. The public source does not prove that Uber migrated specifically from per-service authentication to gateway authentication, so that stronger claim should not be made.

Netflix Zuul

Netflix’s Zuul is an L7 gateway for dynamic routing, monitoring, resiliency, security, and related edge concerns. Netflix’s own Zuul posts list authentication among common edge-service uses, but they do not frame Zuul primarily as a case study in eliminating per-service auth. Treat it as evidence that authentication is a natural edge concern at scale, not as proof of a specific migration story.


The Alternatives

Use these as complementary controls, not as a single replacement for all authentication and authorization logic:

AlternativeWhen to use itWhat it solvesPrincipal trade-off
API Gateway / Edge AuthExternal API clients, public ingress, partner integrations, mixed auth mechanisms at the boundaryCentral JWT/API-key/OAuth2 validation, rate limiting, request shaping, and identity header propagation before traffic reaches servicesDoes not secure east-west service calls by itself; trusted headers require strict network boundaries
Service Mesh mTLSService-to-service traffic inside the cluster, especially across teams or sensitive domainsWorkload identity, automatic mTLS, peer authentication, and proxy-level authorization policyAdds data-plane/control-plane complexity and operational coupling to sidecars or ambient mesh components
OAuth2 ProxyBrowser-facing internal apps that need OIDC login, redirects, cookies, and session handlingDelegates login/session management to a reverse proxy and forwards authenticated identity headersBest for HTTP/browser flows; not a general machine-to-machine authorization system
OPAComplex, auditable, frequently changing authorization rulesSeparates policy decisions from application releases and can run as sidecar, service, ext_authz backend, or admission control via GatekeeperPolicy/data distribution and failure behavior must be designed deliberately
SPIFFE/SPIREMulti-cluster, multi-cloud, or meshless environments that need portable workload identityIssues short-lived workload identities without application-managed shared secretsProvides identity, not business authorization; needs registration and attestation lifecycle management

Option 1: API Gateway (Edge Auth)

An API Gateway sits at the perimeter and handles authentication before a request reaches any downstream service. Services receive pre-validated identity in a trusted header.

What it does: – Validates JWTs, API keys, OAuth2 tokens – Enforces rate limiting per identity – Strips and re-adds Authorization headers as needed – Routes to upstream services with verified identity headers

When to use it: – North-south traffic (external clients → cluster) – Mixed authentication mechanisms (JWT + API key + mTLS) at the Ingress layer – Teams that want to centralize auth policy without rolling out a full service mesh

Tools:

Gravitee.io API Gateway can be deployed on Kubernetes via its Helm chart and integrates with the Kubernetes Gateway API:

helm repo add graviteeio https://helm.gravitee.io
helm install gravitee-apim graviteeio/apim \
  --namespace gravitee \
  --create-namespace \
  --set gateway.replicaCount=2 \
  --set gateway.ingress.enabled=true \
  --set gateway.ingress.hosts[0]=api.example.com

Once deployed, authentication policies are declared on the ApiV4 CRD — no application code involved:

apiVersion: gravitee.io/v1alpha1
kind: ApiV4
metadata:
  name: payment-api
  namespace: gravitee
spec:
  name: "Payment API"
  type: PROXY
  listeners:
    - type: HTTP
      paths:
        - path: /v1/payments
      entrypoints:
        - type: http-proxy
  endpointGroups:
    - name: default
      type: http-proxy
      endpoints:
        - name: upstream
          type: http-proxy
          configuration:
            target: http://payment-service.production.svc.cluster.local:8080
  flows:
    - name: JWT validation
      enabled: true
      request:
        - policy: jwt
          enabled: true
          configuration:
            signature: RSA_RS256
            publicKeyResolver: JWKS_URL
            jwksUrl: https://idp.example.com/.well-known/jwks.json
            checkTokenRevocation: true
            requiredClaims:
              - name: aud
                value: payment-api
        - policy: rate-limit
          enabled: true
          configuration:
            rate:
              limit: 100
              periodTime: 1
              periodTimeUnit: MINUTES

Gravitee’s Kubernetes operator reconciles ApiV4 resources against the gateway, making API policy a first-class GitOps object — versioned, reviewed, and deployed the same way as any other Kubernetes manifest.

Other Kubernetes-native options: Emissary-Ingress (formerly Ambassador) with its AuthService CRD; Traefik with ForwardAuth middleware on IngressRoute resources.

Limitations: API Gateways handle north-south traffic. They don’t address east-west (service-to-service) authentication inside the cluster.


Option 2: Service Mesh (East-West mTLS + Auth)

A service mesh provides mutual TLS between every service pair and enforces authorization policy at the sidecar proxy, without any application code changes.

What it does: – Automatic mTLS between all service-to-service calls – Workload identity via X.509 certificates (SPIFFE SVIDs) – Fine-grained AuthorizationPolicy at the Envoy sidecar – JWT validation at the proxy, not the application

Istio Implementation:

Istio uses Envoy’s ext_authz filter and native RequestAuthentication + AuthorizationPolicy CRDs:

# RequestAuthentication — validate JWTs at the proxy
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: require-jwt
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-service
  jwtRules:
    - issuer: "https://accounts.google.com"
      jwksUri: "https://www.googleapis.com/oauth2/v3/certs"
      audiences:
        - "my-api-audience"
      forwardOriginalToken: false
---
# AuthorizationPolicy — enforce after JWT validation
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payment-service-authz
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/production/sa/order-service"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/payments"]
      when:
        - key: request.auth.claims[scope]
          values: ["payments:write"]

The RequestAuthentication policy tells Envoy how to validate JWTs. The AuthorizationPolicy specifies what authenticated principals are allowed to do. Neither policy lives in application code.

The payment service receives the validated request — or a 401/403 from the proxy, before the request touches application code.

Linkerd:

Linkerd provides automatic mTLS with SPIFFE-compliant workload identity. Its policy model is simpler than Istio but sufficient for most service-to-service auth requirements:

apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata:
  name: payment-server
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: payment-service
  port: 8080
---
apiVersion: policy.linkerd.io/v1beta3
kind: ServerAuthorization
metadata:
  name: order-to-payment
  namespace: production
spec:
  server:
    name: payment-server
  client:
    meshTLS:
      serviceAccounts:
        - name: order-service

This is mutual TLS + SPIFFE-based identity, enforced at the proxy. The application doesn’t implement it; the mesh does.

Istio + Envoy External Authorization:

For more complex policy (e.g., OPA integration), Envoy’s ext_authz filter delegates authorization to an external service:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz-filter
  namespace: production
spec:
  workloadSelector:
    labels:
      app: payment-service
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ext_authz
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
            grpc_service:
              envoy_grpc:
                cluster_name: outbound|9191||opa.production.svc.cluster.local
            timeout: 0.5s
            failure_mode_allow: false

The CNCF TAG Security paper on microservices security documents this architecture as the reference pattern for production Kubernetes environments.


Option 3: OAuth2 Proxy (Delegated Auth for HTTP)

OAuth2 Proxy is a reverse proxy that authenticates requests against an OAuth2/OIDC provider and passes validated identity downstream. With 14,000+ GitHub stars and active maintenance, it is the most widely deployed solution for this pattern in Kubernetes.

What it does: – Sits in front of one or more upstream services – Redirects unauthenticated requests to an OIDC provider (Keycloak, Dex, Google, GitHub, etc.) – Validates tokens, manages sessions, handles refresh – Passes X-Auth-Request-User, X-Auth-Request-Email, X-Auth-Request-Groups headers downstream

Kubernetes deployment with Nginx Ingress:

# OAuth2 Proxy deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oauth2-proxy
  namespace: auth
spec:
  replicas: 2
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
        - name: oauth2-proxy
          image: quay.io/oauth2-proxy/oauth2-proxy:v7.6.0
          args:
            - --provider=oidc
            - --oidc-issuer-url=https://keycloak.example.com/realms/myrealm
            - --client-id=$(CLIENT_ID)
            - --client-secret=$(CLIENT_SECRET)
            - --cookie-secret=$(COOKIE_SECRET)
            - --http-address=0.0.0.0:4180
            - --reverse-proxy=true
            - --upstream=static://202
            - --email-domain=*
            - --set-xauthrequest=true
            - --cookie-secure=true
            - --skip-provider-button=true
          env:
            - name: CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: oauth2-proxy-secrets
                  key: client-id
            - name: CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: oauth2-proxy-secrets
                  key: client-secret
            - name: COOKIE_SECRET
              valueFrom:
                secretKeyRef:
                  name: oauth2-proxy-secrets
                  key: cookie-secret
---
# Ingress annotation to protect a service with OAuth2 Proxy
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: protected-service
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "https://oauth2-proxy.example.com/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://oauth2-proxy.example.com/oauth2/start?rd=$escaped_request_uri"
    nginx.ingress.kubernetes.io/auth-response-headers: "X-Auth-Request-User,X-Auth-Request-Email,X-Auth-Request-Groups"
spec:
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: protected-service
                port:
                  number: 8080

The upstream service receives X-Auth-Request-User and X-Auth-Request-Groups as trusted headers — it never sees a token, never validates a signature, never imports a JWT library.

When to use OAuth2 Proxy vs a service mesh: OAuth2 Proxy handles north-south browser-facing traffic with session management (login flows, redirects, cookies). A service mesh handles east-west machine-to-machine auth. They are complementary, not alternatives.


Option 4: Open Policy Agent (OPA / OPAL)

OPA decouples policy from code entirely. Authorization logic is written in Rego and evaluated by OPA as a sidecar or as a centralized service. Applications query OPA for allow/deny decisions.

# Rego policy — payment service authorization
package payments.authz

import future.keywords.if
import future.keywords.in

default allow := false

allow if {
    input.method == "POST"
    input.path == "/v1/payments"
    "payments:write" in input.token.scope
    input.token.iss == "https://accounts.example.com"
}

allow if {
    input.method == "GET"
    startswith(input.path, "/v1/payments/")
    "payments:read" in input.token.scope
}

Application code becomes:

// The ONLY auth code in the application
func authMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        input := map[string]interface{}{
            "method": r.Method,
            "path":   r.URL.Path,
            "token":  extractToken(r),
        }
        
        result, err := opaClient.Decision(r.Context(), "payments/authz", input)
        if err != nil || !result.Allow {
            http.Error(w, "Forbidden", http.StatusForbidden)
            return
        }
        next.ServeHTTP(w, r)
    })
}

OPAL (Open Policy Administration Layer) adds real-time policy and data updates to OPA deployments — policy changes propagate to all OPA instances within seconds without redeployment.

Kubernetes deployment patterns for OPA:

As a sidecar — OPA runs in the same Pod as the application, evaluating policy over a local socket. Zero network hop, no external dependency:

# Pod template fragment
spec:
  containers:
    - name: payment-service
      image: payment-service:latest
    - name: opa
      image: openpolicyagent/opa:0.63.0
      args:
        - run
        - --server
        - --addr=localhost:8181
        - /policy
      volumeMounts:
        - name: opa-policy
          mountPath: /policy
          readOnly: true
  volumes:
    - name: opa-policy
      configMap:
        name: payment-policy

As a centralized service with Envoy ext_authz — OPA exposes a gRPC endpoint that Istio’s Envoy sidecar calls for every request. Policy is enforced at the proxy, before the application receives the request. This is the pattern used alongside Istio’s EnvoyFilter shown in the service mesh section above.

As OPA GatekeeperOPA Gatekeeper runs as a Kubernetes admission webhook and enforces policies at deploy time, not at runtime. It’s the right tool for preventing misconfigured workloads from being deployed — for example, rejecting any Pod spec that sets hostNetwork: true or defines auth-related environment variables directly. This is complementary to runtime auth enforcement.

OPA is used at scale by Atlassian, Goldman Sachs, Netflix, Chef, and many others, documented in OPA’s production deployments. The CNCF OPA project graduated in 2021.


Option 5: SPIFFE/SPIRE (Workload Identity)

SPIFFE (Secure Production Identity Framework for Everyone) and SPIRE solve the problem of how workloads prove their identity without distributing secrets.

SPIRE issues short-lived X.509 SVIDs (SPIFFE Verifiable Identity Documents) to workloads. Each SVID encodes a SPIFFE URI:

spiffe://example.org/ns/production/sa/payment-service

Services authenticate each other using mTLS with these certificates. No JWT library. No shared secret. No secret distribution problem. Certificate rotation happens automatically every few hours.

# SPIRE Agent DaemonSet on Kubernetes
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: spire-agent
  namespace: spire
  labels:
    app: spire-agent
spec:
  selector:
    matchLabels:
      app: spire-agent
  template:
    metadata:
      labels:
        app: spire-agent
    spec:
      serviceAccountName: spire-agent
      hostPID: true
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: spire-agent
          image: ghcr.io/spiffe/spire-agent:1.9.0
          args: ["-config", "/run/spire/config/agent.conf"]
          volumeMounts:
            - name: spire-config
              mountPath: /run/spire/config
              readOnly: true
            - name: spire-agent-socket
              mountPath: /run/spire/sockets
              readOnly: false
      volumes:
        - name: spire-config
          configMap:
            name: spire-agent
        - name: spire-agent-socket
          hostPath:
            path: /run/spire/sockets
            type: DirectoryOrCreate

SPIFFE is the foundation of Istio’s workload identity model. Linkerd implements SPIFFE-compatible SVIDs. If you’re using a service mesh, you already have SPIFFE — the mesh uses it transparently.

Standalone SPIRE is appropriate for environments without a service mesh, or for multi-cluster/multi-cloud scenarios where a consistent workload identity layer is needed across boundaries.

SPIFFE/SPIRE graduated from the CNCF sandbox to incubating in 2019 and is deployed at Uber, Bloomberg, ByteDance, and Anthem.


Combining the Layers

These tools are not mutually exclusive — they address different traffic patterns and different problems:

LayerToolAddresses
Edge (north-south)API Gateway (Gravitee, Emissary, AWS APIGW)External clients → cluster
Browser sessionsOAuth2 ProxyBrowser-facing apps with login flows
East-west mTLSService Mesh (Istio/Linkerd) + SPIFFEService-to-service identity
PolicyOPA/OPALFine-grained, auditable authorization
Workload identitySPIREMulti-cloud/multi-cluster identity

A production deployment at reasonable scale looks like:

  1. External traffic hits an API Gateway or Ingress controller with OAuth2 Proxy
  2. The gateway validates the token, strips it, and forwards identity headers to the upstream service
  3. Inside the cluster, all service-to-service calls are mTLS via a service mesh, using SPIFFE workload identity
  4. Authorization decisions (beyond identity) are delegated to OPA
  5. Application code contains zero JWT validation, zero session management, zero auth library imports

Migration Path

If you already have auth embedded in services, migration doesn’t require a big bang rewrite.

Phase 1: Introduce the Gateway

Deploy an API Gateway or OAuth2 Proxy at the edge. Initially, services continue to validate tokens themselves as a backup — the gateway validates first. Use this phase to verify the gateway’s behavior and build confidence.

Phase 2: Trust the Gateway

Add a service-level feature flag: if a trusted X-Auth-Request-User header is present (set by the gateway), skip internal JWT validation. This decouples service auth from gateway rollout.

Phase 3: Remove In-Service Auth

Once all entry points are covered by the gateway and you have confidence in its reliability, remove the auth code from services. This is the step that actually reduces your attack surface.

Phase 4: Add East-West (Optional)

If east-west service-to-service calls exist and carry sensitive data, introduce a service mesh for mTLS. This is a separate effort from gateway auth and can proceed independently.


Decision Framework

External traffic entering the cluster (Ingress / Gateway API)?
├── Browser-facing app with login flow → OAuth2 Proxy (Nginx Ingress annotations)
└── API clients with tokens → API Gateway (Gravitee ApiV4 / Emissary AuthService)

Service-to-service calls inside the cluster (east-west)?
├── mTLS + identity sufficient → Service Mesh (Istio/Linkerd)
├── Identity across multiple clusters → SPIFFE/SPIRE standalone
└── Istio + fine-grained policy → RequestAuthentication + OPA via ext_authz

Complex, auditable authorization logic?
└── OPA as sidecar or ext_authz (runtime) + OPA Gatekeeper (admission)

Preventing misconfigured workloads from being deployed?
└── OPA Gatekeeper admission webhook

What to Keep in Application Code

Not everything should be removed. The correct model is:

  • Remove: JWT signature verification, token parsing, OAuth2 flows, session management
  • Keep: Business-level authorization (“can this user edit this specific resource?”), assuming identity is provided by infrastructure
  • Keep: Authorization errors surfaced correctly (403 vs 401, meaningful error bodies)
  • Keep: Structured logging of authorization decisions for audit trails

The application receives an authenticated identity from infrastructure. What the application does with that identity — which records to show, which operations to allow based on ownership — is correctly application logic.


Audit Checklist: Moving Auth Out of Containers

Use this as a practical exit checklist for the anti-pattern:

  • Inventory every service that imports JWT, OAuth2, OIDC, session, or custom RBAC libraries.
  • Classify each entry point as north-south, browser session, east-west service call, deploy-time admission policy, or business authorization.
  • Put one enforcing control in front of each class: API Gateway or OAuth2 Proxy for ingress, service mesh mTLS for east-west, OPA for shared policy, and SPIFFE/SPIRE for portable workload identity.
  • Pin JWT issuers, audiences, algorithms, and JWKS sources in infrastructure policy; do not let application code infer them from token headers.
  • Strip client-supplied identity headers at the edge and re-add trusted identity headers only after verification.
  • Define failure behavior explicitly: fail closed for authentication and authorization, and document any temporary fail-open exception with an owner and expiry date.
  • Remove in-service token verification only after every ingress path is covered, logs prove the infrastructure control is enforcing, and rollback has been tested.
  • Keep resource-level business authorization in application code, but feed it an identity established by infrastructure.

References

Standards and Frameworks – NIST SP 800-204: Security Strategies for MicroservicesNIST SP 800-204A: Building Secure Microservices-based Applications Using Service-Mesh ArchitectureNIST SP 800-207: Zero Trust ArchitectureOWASP Kubernetes Security Cheat SheetOWASP Microservices Security Cheat SheetCNCF Cloud Native Security Whitepaper v2RFC 7519: JSON Web Token (JWT)RFC 8725: JSON Web Token Best Current Practices

Vulnerabilities and Attack Classes – CVE-2022-21449: Java Psychic Signatures (ECDSA bypass)analysis by Neil MaddenCVE-2023-2728: Kubernetes mountable secrets policy bypassCVE-2023-2727: Kubernetes ImagePolicyWebhook bypassAuth0: Critical Vulnerabilities in JSON Web Token Libraries (alg:none, RS/HS confusion)PortSwigger Web Security Academy: JWT attacksjwt.io: Debugger and library reference

Tools and Projects – OAuth2 Proxy (GitHub — 14k+ stars)OAuth2 Proxy DocumentationGravitee.io API Gateway — JWT PolicyGravitee Kubernetes OperatorGravitee Helm ChartEmissary-Ingress AuthServiceOPA GatekeeperIstio Security: RequestAuthentication and AuthorizationPolicyEnvoy External Authorization FilterLinkerd Server PolicyOpen Policy AgentOPAL — Open Policy Administration LayerSPIFFE — Secure Production Identity Framework for EveryoneSPIRE — SPIFFE Runtime EnvironmentNetflix Zuul (GitHub)Traefik ForwardAuth Middleware

Architecture and Industry Context – Google BeyondCorp: A New Approach to Enterprise SecurityGoogle BeyondCorp Research Paper (USENIX ;login:)Netflix Tech Blog: Zuul 2 — The Netflix Journey to Asynchronous, Non-Blocking SystemsSPIFFE/SPIRE CNCF Graduation AnnouncementOPA CNCF GraduationInfoQ: Microservices Authentication and Authorization Anti-PatternsAWS re:Invent: Zero Trust Networking on AWSIstio Service Mesh Security ArchitectureThe CNCF TAG Security Microservices Security Paper


Article reflects tooling as of 2026: Kubernetes 1.29+, Istio 1.21+, Linkerd 2.15+, OPA 0.63+ / Gatekeeper 3.16+, SPIRE 1.9+, OAuth2 Proxy 7.6+, Gravitee APIM 4.x.

Sources

Kubernetes Security Best Practices: 2026 Production Hardening Guide

Kubernetes Security Best Practices: 2026 Production Hardening Guide

Kubernetes security is not a single feature you enable — it is a layered discipline that spans the control plane, workloads, networking, supply chain, and runtime. Get one layer wrong and the others rarely save you. This guide covers the controls that matter most in production, why each one exists, and how to implement them without breaking your cluster — plus a prioritized roadmap so you know what to do in your first week, not just an undifferentiated list of “best practices.”

Related reading: the authentication-inside-containers anti-pattern.

Let me start with the part most hardening guides skip: what an actual attack looks like.

Anatomy of a Real Kubernetes Attack Chain

Abstract advice (“apply least privilege”) doesn’t land until you’ve seen how a single misconfiguration cascades. Here is a realistic chain — every step maps to a documented technique in the MITRE ATT&CK for Containers matrix. If you haven’t seen it before, ATT&CK is an industry-standard, openly maintained knowledge base of real-world adversary behavior: a catalogue of how attackers actually operate, organized by goal (initial access, credential access, lateral movement, and so on). It’s the common language security teams use to describe and defend against attacks.

  1. Initial access. An application pod runs a vulnerable image — say, an unpatched dependency with a remote code execution (RCE) flaw, a bug that lets an attacker run arbitrary code on the host process. The attacker gets code execution inside the container. So far, container isolation should contain the blast radius.
  2. Credential access. The pod has automountServiceAccountToken: true (the default). The attacker reads /var/run/secrets/kubernetes.io/serviceaccount/token — a valid API credential, handed to them for free.
  3. Discovery. Using that token, the attacker queries the API server. The ServiceAccount was bound to a convenient cluster-admin role “to unblock a deploy.” Now they can list every Secret in every namespace.
  4. Lateral movement. They read database credentials, cloud provider keys, and other ServiceAccount tokens from Secrets. The flat pod network (no NetworkPolicies) lets them reach internal services directly.
  5. Privilege escalation / escape. They schedule a privileged pod with hostPID and the host filesystem mounted, then break out to the node. From the node, they reach the kubelet and other tenants’ workloads.
  6. Impact. Crypto-mining, data exfiltration, or ransomware across the cluster.

Notice that steps 2 through 5 each had a one-line fix: disable token automount, scope the RBAC, encrypt Secrets / use an external store, apply default-deny NetworkPolicies, enforce Pod Security. Defense in depth means an attacker has to defeat every layer — and most attackers give up when the easy chain breaks. The rest of this guide is those layers, ordered by how much they shrink that chain.

The Kubernetes Attack Surface

Before hardening anything, understand what you are protecting. A Kubernetes cluster has several distinct attack surfaces:

  • API server — The central control plane. Any entity that can reach it with valid credentials can read cluster state, modify workloads, or escalate privileges.
  • etcd — Stores all cluster state in plain text, including Secrets. Direct etcd access is equivalent to root on every node.
  • Nodes — A compromised node can access all Secrets mounted on pods running on it, access the kubelet API, and potentially escape to the hypervisor.
  • Pods — Privileged pods, host-network pods, and pods with excessive capabilities can break container isolation.
  • Supply chain — Malicious images, compromised registries, and unsigned artifacts can introduce attacker-controlled code into your cluster.
  • RBAC — Overly permissive roles allow lateral movement and privilege escalation once an attacker gains any foothold.

Prioritize based on your threat model — a public-facing multi-tenant cluster needs all of these; an internal development cluster can relax some.

The First-Week Hardening Roadmap (Prioritized)

If you inherited a cluster with nothing in place, do not try to do everything at once. Order matters — some controls give huge risk reduction for minimal effort and zero breakage risk, others need careful rollout. This is the sequence I use:

DayControlRisk reductionBreakage risk
1Audit RBAC, remove stray cluster-admin, disable unused SA token automountHighLow
1Enable API server audit loggingMedium (visibility)None
2Pod Security Admission in warn + audit mode (all namespaces)HighNone (warn only)
3Deploy image scanning in CI (Trivy/Grype), fail on CriticalHighLow
4NetworkPolicies in audit-style rollout: default-deny in one namespace firstHighMedium — test DNS!
5Enable etcd encryption at rest / move Secrets to external storeHighLow
6Flip Pod Security Admission to enforce: baseline, then restricted per namespaceHighMedium
7Deploy runtime detection (Falco) + continuous scanning (Trivy Operator)MediumNone

The single most important idea: roll out enforcing controls in observation mode first (warn/audit for Pod Security, default-deny NetworkPolicies in one namespace). You want to discover what breaks in a dashboard, not in an incident.

Tools to automate and report each step

You don’t have to do any of this by hand. Each step has tooling that both applies the control and reports on its state, so you can wire it into CI or a recurring job:

1. RBAC: Least Privilege from Day One

Role-Based Access Control is Kubernetes’ primary authorization mechanism. Most clusters fail at RBAC not because it is misconfigured, but because it is over-permissive by default and nobody reviews it systematically.

Common RBAC Mistakes

  • Binding to cluster-admin for convenience. Almost no workload needs cluster-admin. Use namespaced roles wherever possible.
  • Using * verbs or resources in roles. Wildcard permissions are almost always broader than intended.
  • Not auditing ServiceAccount token usage. Every pod gets a ServiceAccount. Custom workloads often get over-permissive SAs.
  • Forgetting automountServiceAccountToken: false. If a workload does not need to talk to the Kubernetes API, disable token mounting entirely — this single setting breaks step 2 of the attack chain above.

Practical RBAC Patterns

For a workload that only needs to read ConfigMaps in its own namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: configmap-reader
  namespace: my-app
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-app-configmap-reader
  namespace: my-app
subjects:
- kind: ServiceAccount
  name: my-app
  namespace: my-app
roleRef:
  kind: Role
  name: configmap-reader
  apiGroup: rbac.authorization.k8s.io

And disable token automount on the workload that doesn’t call the API at all:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: my-app
automountServiceAccountToken: false

Audit existing RBAC with kubectl-who-can or rbac-tool to find overly permissive bindings before attackers do. A useful one-liner: list every subject that can read Secrets cluster-wide with kubectl who-can get secrets.

2. Pod Security Standards (and Migrating off PodSecurityPolicy)

PodSecurityPolicy was deprecated in Kubernetes 1.21 and removed in 1.25. Its replacement is Pod Security Admission (PSA), a built-in admission controller that enforces one of three Pod Security Standards profiles at the namespace level:

  • Privileged — No restrictions. For system components only.
  • Baseline — Prevents the most critical privilege escalations: privileged containers, hostPID, hostIPC, hostNetwork, dangerous capabilities.
  • Restricted — Enforces current hardening best practices. Requires running as non-root, dropping all capabilities, and using a restricted seccomp profile.

Enable enforcement at the namespace level with labels:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.30
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.30
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.30

A pod that runs as root or requests host-network in a namespace enforcing restricted will be rejected at admission. The warn and audit modes let you test before enforcing. For a full walkthrough of how PSA evaluates pods and how to roll it out, see my guide on understanding Pod Security Admission.

Migrating from PodSecurityPolicy to PSA

If you’re still on a cluster that used PSP, the migration path is:

  1. Map your PSPs to the closest PSA level. Most “restricted” PSPs map to restricted; permissive ones to baseline. The official pspmigrator tool can suggest mappings.
  2. Label every namespace in warn/audit mode matching that level — no enforcement yet.
  3. Watch the audit logs and warnings for a release cycle. Fix the workloads that would be rejected (add securityContext, drop capabilities).
  4. Flip to enforce namespace by namespace, starting with the least critical.

PSA is intentionally coarse-grained — three levels, namespace-scoped. For anything finer (per-team registries, required labels, custom mutation), you need a policy engine, which is the next section.

3. Policy Engines: Kyverno vs OPA Gatekeeper

Once you outgrow PSA’s three levels, you need an admission policy engine. The two standards are Kyverno and OPA Gatekeeper, and choosing between them is one of the most common platform decisions.

KyvernoOPA Gatekeeper
Policy languageYAML (Kubernetes-native)Rego (purpose-built DSL)
Learning curveLow — looks like other manifestsSteep — Rego is its own paradigm
Mutation supportYes, first-classLimited
Image verification (Cosign)Built-inVia external data
Best whenTeam wants fast adoption, K8s-onlyTeam already runs OPA across the stack

For most teams without existing Rego expertise, Kyverno is significantly faster to adopt and maintain. A Kyverno policy to require all images come from your private registry:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-registries
    match:
      any:
      - resources:
          kinds: ["Pod"]
    validate:
      message: "Images must come from registry.company.com"
      pattern:
        spec:
          containers:
          - image: "registry.company.com/*"

Both integrate cleanly with GitOps — store policies in Git, apply via Argo CD or Flux, and you get an auditable history of every policy change. I’ve written several deep dives on this: Kyverno: enforcing standard and custom policies, extending Kyverno with custom rules, and running the Kyverno CLI in CI/CD with GitHub Actions — or browse everything under the policies tag.

4. Network Policies: Micro-Segmentation

By default, every pod in a Kubernetes cluster can communicate with every other pod across all namespaces. This flat network model gives attackers unrestricted lateral movement once they compromise any workload (step 4 of the attack chain).

Network Policies define L3/L4 allow-rules for pod-to-pod communication. They are enforced by your CNI (Container Network Interface) plugin (Calico, Cilium, Weave — not Flannel, which does not support NetworkPolicy).

Default Deny Pattern

Start by denying all ingress and egress in a namespace, then open only what is explicitly needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Then allow specific traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - protocol: TCP
      port: 5432

The DNS Trap (the #1 reason default-deny “breaks everything”)

The most common NetworkPolicy support ticket: “I applied default-deny and the whole namespace stopped working.” The cause is almost always DNS. A default-deny egress policy blocks the pod from reaching kube-dns, so every name resolution fails and applications appear to hang or crash-loop.

Always pair default-deny egress with an explicit DNS allow rule:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Roll default-deny out in one namespace first, confirm DNS and required egress work, then expand. Tools like Cilium’s Hubble or Calico’s flow logs make it much easier to see exactly which flows you need to allow.

5. Secrets Management

Kubernetes Secrets are base64-encoded, not encrypted. They are stored in etcd in plain text by default. Anyone with get permission on Secrets can read them. This is not a vulnerability — it is a design decision that puts the responsibility on you to:

  • Enable encryption at rest for etcd. Configure EncryptionConfiguration with an AES-CBC or AES-GCM provider so Secrets are encrypted before being written to etcd.
  • Use external secret stores. HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault with the External Secrets Operator means actual secret values never live in Kubernetes at all.
  • Restrict Secret RBAC aggressively. Never give list on Secrets cluster-wide — it returns all values. Use get on named resources where possible.
  • Avoid environment variables for secrets. Prefer volume mounts. Env vars are visible in pod inspect output and can leak through application logging.
# etcd encryption at rest - in kube-apiserver config
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-32-byte-key>
  - identity: {}

For the full external-store pattern, see the guide on injecting secrets into pods with HashiCorp Vault.

6. Image Security and Supply Chain

Your runtime security posture is only as good as the images you run. A compromised image from a public registry bypasses every runtime control you have.

Scan images in CI

Use Trivy, Grype, or Snyk to scan images as part of CI. Block deployments of images with critical CVEs (Common Vulnerabilities and Exposures — publicly catalogued security flaws). I’ve covered the practical side of this in scanning Docker images with Trivy, scanning your images locally before they ship, and a broader roundup of open-source development security tools.

# In your CI pipeline
trivy image --exit-code 1 --severity CRITICAL your-image:tag

Use a private registry with admission control

Only allow images from your private registry using an admission webhook (Kyverno, OPA Gatekeeper) — the policy in section 3 does exactly this. It prevents developers from running arbitrary public images in production.

Use distroless or minimal base images

Distroless images contain only the application and its runtime dependencies — no shell, no package manager, no debugging tools. This drastically reduces both the attack surface and the CVE count. Google’s distroless images are available for Java, Node.js, Python, and Go. (Related: debugging distroless containers when you do need to inspect one.)

Sign and verify images (and the SLSA angle)

Cosign (from the Sigstore project) lets you sign container images and verify signatures at admission time using Kyverno or Connaisseur. This prevents image-substitution attacks where an attacker replaces a legitimate image in your registry.

If you’re being asked about supply-chain compliance, the framework to know is SLSA (Supply-chain Levels for Software Artifacts). The practical progression: SLSA L1 = you have a build provenance document; L2 = it’s signed and the build is hosted; L3 = the build is hardened and non-falsifiable. Generating provenance with your CI (GitHub Actions has native SLSA generators) and verifying it at admission with Cosign + Kyverno gets you most of the way to L2/L3 without a platform rebuild.

7. Runtime Security

Runtime security detects and responds to malicious activity after a container is running. The primary tool is Falco — a CNCF project that uses eBPF (extended Berkeley Packet Filter — a Linux kernel technology for running sandboxed observability programs) to monitor system calls and raise alerts when containers behave unexpectedly.

Default Falco rules catch common attack patterns:

  • Shell spawned in a container
  • Network connection to an unexpected IP
  • Write to a sensitive file path (/etc/passwd, /etc/shadow)
  • Privilege escalation via setuid binaries
  • Container drift (new executable files written at runtime)

Combine Falco with seccomp profiles to restrict the system calls a container can make at the kernel level. The RuntimeDefault seccomp profile (a default option since Kubernetes 1.27) blocks 300+ system calls that containers virtually never need.

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 65534
      capabilities:
        drop: ["ALL"]

These four securityContext settings together (allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, runAsNonRoot: true, capabilities.drop: ALL) make container escape significantly harder and satisfy the Kubernetes Restricted pod security standard. They directly close step 5 of the attack chain.

8. API Server Hardening

The API server is the most critical component to harden. Key settings:

  • Disable anonymous authentication. --anonymous-auth=false ensures every request is authenticated.
  • Enable audit logging. Log all API server requests to a file or webhook. Without audit logs, you cannot investigate incidents or detect RBAC abuse.
  • Restrict admission plugins. Ensure NodeRestriction is enabled — it prevents node kubelets from modifying objects outside their own node.
  • Do not expose the API server to the internet. Use a VPN, bastion host, or private endpoint. If you must expose it, restrict access by IP.
# Minimal audit policy - log all requests at metadata level,
# and full request body for sensitive resources
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets", "configmaps"]
- level: Metadata
  omitStages: ["RequestReceived"]

9. etcd Security

etcd stores all cluster state. Treat it as sensitive as your production database:

  • Enable TLS for all etcd communication — both peer (etcd-to-etcd) and client (apiserver-to-etcd) with mutual TLS.
  • Restrict network access to etcd. It should only be reachable by the API server. Use firewall rules or security groups.
  • Enable encryption at rest (see Secrets section).
  • Back up etcd regularly. A snapshot is a complete copy of all cluster state, including all Secrets. Encrypt backups and store them separately from the cluster.

10. Multi-Tenancy Isolation

If multiple teams or customers share a cluster, namespace boundaries alone are not a security boundary — they’re an organizational one. Hardening multi-tenant clusters adds requirements on top of everything above:

  • Namespace-per-tenant with ResourceQuotas and LimitRanges to prevent noisy-neighbor and resource-exhaustion DoS.
  • NetworkPolicies that deny cross-namespace traffic by default, so tenant A cannot reach tenant B’s pods.
  • A policy engine enforcing per-tenant rules (allowed registries, required labels, no hostPath).
  • Separate node pools for untrusted workloads, or a sandboxed runtime (gVisor, Kata Containers) when you run genuinely untrusted code.

For hard multi-tenancy (untrusted tenants), the honest answer is that vanilla namespaces aren’t enough — consider virtual clusters (vCluster) or separate clusters entirely. Soft multi-tenancy (trusted internal teams) is well served by the controls in this guide.

11. Benchmarks and Continuous Posture

CIS Kubernetes Benchmark

The CIS Kubernetes Benchmark is a comprehensive checklist covering the control plane, nodes, and workloads. Running kube-bench gives you a scored assessment:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs $(kubectl get pods -l app=kube-bench -o name)

kube-bench outputs PASS/FAIL/WARN for each control with remediation guidance. Run it after initial cluster setup and after major configuration changes.

Continuous scanning with Trivy Operator / Kubescape

Kubescape and the Trivy Operator provide continuous security scanning of live cluster state — not just a one-time audit. They check workloads against NSA/CISA hardening guidelines, the MITRE ATT&CK framework, and the CIS benchmark in real time.

helm repo add aquasecurity https://aquasecurity.github.io/helm-charts/
helm install trivy-operator aquasecurity/trivy-operator 
  --namespace trivy-system 
  --create-namespace 
  --set="trivy.ignoreUnfixed=true"

Trivy Operator creates VulnerabilityReport, ConfigAuditReport, and RbacAssessmentReport custom resources alongside each workload. Scrape them with Prometheus and build a security dashboard in Grafana.

Security Hardening Checklist

  • ✅ RBAC reviewed — no wildcard roles, no unnecessary cluster-admin bindings
  • ✅ ServiceAccount token automount disabled for workloads that do not need API access
  • ✅ Pod Security Standards enforced at namespace level (at least Baseline, Restricted where possible)
  • ✅ Policy engine (Kyverno/Gatekeeper) enforcing registry, label, and mutation rules
  • ✅ Network policies deployed — default deny with explicit allows (including DNS!)
  • ✅ Secrets encrypted at rest in etcd or moved to an external store
  • ✅ Images scanned in CI — no critical CVEs in production
  • ✅ Private registry enforced via admission control
  • ✅ Image signing + verification (Cosign) and build provenance (SLSA)
  • ✅ Container securityContext hardened (non-root, read-only fs, no capabilities)
  • ✅ seccomp RuntimeDefault profile enabled
  • ✅ API server audit logging enabled, anonymous auth disabled
  • ✅ etcd TLS and network access restricted
  • ✅ Multi-tenancy isolation (quotas, cross-namespace deny) if shared
  • ✅ kube-bench run and critical/high findings remediated
  • ✅ Runtime security (Falco) deployed and alerts routed to on-call
  • ✅ Continuous scanning (Trivy Operator or Kubescape) deployed

FAQ

Where do I start if my cluster has no security controls today?

Follow the first-week roadmap above. The short version: audit RBAC (revoke stray cluster-admin), enable Pod Security Admission in warn mode on all namespaces, and deploy image scanning + Trivy Operator. These give immediate visibility and stop the most common privilege escalations without breaking anything.

Does enabling Network Policies break DNS resolution?

Yes — this is the single most common failure. A default-deny egress policy blocks pods from reaching kube-dns, so name resolution fails. Add an egress rule allowing UDP and TCP port 53 to the kube-system namespace whenever you apply default-deny (see the DNS allow policy above).

Should I use OPA Gatekeeper or Kyverno?

Both enforce admission policies. Kyverno is Kubernetes-native (policies are YAML) while Gatekeeper uses Rego. For teams without Rego expertise, Kyverno is faster to adopt and supports mutation and Cosign verification out of the box. Choose Gatekeeper if you already run OPA elsewhere and want one policy language across your stack.

What replaced PodSecurityPolicy?

Pod Security Admission (PSA), built into Kubernetes since 1.25. It enforces three profiles (privileged/baseline/restricted) via namespace labels. For finer-grained control than PSA’s three levels, add Kyverno or Gatekeeper.

Is Kubernetes certified for PCI-DSS or SOC 2?

Kubernetes itself is not certified — your configuration and the controls you implement determine compliance. The CIS Kubernetes Benchmark maps to many PCI-DSS and SOC 2 requirements. Managed offerings (EKS, GKE, AKS) carry their own compliance certifications for the underlying infrastructure.

How often should I update Kubernetes for security patches?

Apply a patch release within 30 days for High/Critical CVEs. Minor version upgrades (e.g., 1.30 → 1.31) should happen within the support window — Kubernetes maintains the last three minor versions. Falling more than one minor behind means running without patches for a growing subset of the codebase.

Are namespaces a security boundary?

No. Namespaces are an organizational boundary. Real isolation between tenants requires NetworkPolicies, ResourceQuotas, a policy engine, and — for untrusted workloads — sandboxed runtimes (gVisor/Kata) or separate clusters.


For a deeper look at how security fits into the broader Kubernetes platform architecture, see the Kubernetes architecture patterns guide and the guide on building a security-first Kubernetes culture.

Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each

Developer inspecting a distroless container with magnifying glass

The container works fine in CI. It deploys successfully to staging. Then something goes wrong in production and you type the command you always type: kubectl exec -it my-pod -- /bin/bash. The response is immediate: OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory.

You try /bin/sh. Same error. You try ls. Same error. The container image is distroless — it ships only your application binary and its runtime dependencies, with no shell, no package manager, no debugging tools of any kind. This is intentional and correct from a security standpoint. It is also a significant operational challenge the first time you face it in production.

This article covers every practical technique for debugging distroless containers in Kubernetes: kubectl debug with ephemeral containers (the standard approach), pod copy strategy (for Kubernetes versions without ephemeral container support, or when you need to modify the running pod spec), debug image variants (the pragmatic developer shortcut), cdebug (a purpose-built tool that simplifies the process), and node-level debugging (the last resort with the most power). For each technique I will explain what it can and cannot do, what Kubernetes version or RBAC permissions it requires, and in which scenario — developer in local, platform engineer in staging, ops in production — it is the appropriate choice.

Why Distroless Breaks the Normal Debugging Workflow

Traditional container debugging assumes you can exec into the container and use shell tools: ps, netstat, strace, curl, a text editor. Distroless images remove all of this by design. The Google distroless project, Chainguard’s Wolfi-based images, and the broader minimal image ecosystem deliberately exclude everything that is not required to run the application. The result is a dramatically smaller attack surface: no shell means no RCE via shell injection, no package manager means no easy escalation path, fewer binaries means fewer CVEs in the image scan.

The tradeoff is operational: when something goes wrong, you cannot use the tools that the process itself is not allowed to run. A Java application in gcr.io/distroless/java17-debian12 has the JRE and nothing else. A Go binary compiled with CGO disabled and shipped in gcr.io/distroless/static-debian12 has literally only the binary and the necessary CA certificates and timezone data. There is no wget to download a debug binary, no apt to install one, no bash to run a script.

Kubernetes solves this at the platform level with ephemeral containers, added as stable in Kubernetes 1.25. The principle is that a debug container — which can have a full shell and any tools you want — can be injected into a running pod and share its process namespace, network namespace, and filesystem mounts without modifying the original container or restarting the pod.

Option 1: kubectl debug with Ephemeral Containers

Ephemeral containers are the canonical solution. Since Kubernetes 1.25 (stable), kubectl debug can inject a temporary container into a running pod. The container shares the target pod’s network namespace by default, and with --target it can also share the process namespace of a specific container, allowing you to inspect its running processes and open file descriptors.

The basic invocation is:

kubectl debug -it my-pod \
  --image=busybox:latest \
  --target=my-container

The --target flag is the critical piece. Without it, the ephemeral container gets its own process namespace. With it, it shares the process namespace of the specified container — meaning you can run ps aux and see the application’s processes, use ls -la /proc/<pid>/fd to inspect open file descriptors, and read the application’s environment via cat /proc/<pid>/environ.

For a more capable debug environment, replace busybox with a richer image:

kubectl debug -it my-pod \
  --image=nicolaka/netshoot \
  --target=my-container

nicolaka/netshoot includes tcpdump, curl, dig, nmap, ss, iperf3, and dozens of other network diagnostic tools, making it the standard choice for network debugging scenarios.

What You Can and Cannot Do

Ephemeral containers share the pod’s network namespace and, when --target is used, the process namespace. This gives you:

  • Full visibility into the application’s network traffic from inside the pod (tcpdump, ss, netstat)
  • Process inspection via /proc/<pid> — open files, memory maps, environment variables, CPU/memory usage
  • Access to the pod’s DNS resolution context — exactly the same /etc/resolv.conf the application sees
  • Ability to make outbound network calls from the same network namespace (testing service endpoints, DNS resolution)

What you do not get with ephemeral containers:

  • Access to the application container’s filesystem. The ephemeral container has its own root filesystem. You cannot cat /app/config.yaml from the application container’s filesystem unless you access it via /proc/<pid>/root/.
  • Ability to remove the container once added. Ephemeral containers are permanent until the pod is deleted. This is by design — the Kubernetes API does not allow removing them after creation.
  • Volume mount modifications via CLI. You cannot add volume mounts to an ephemeral container via kubectl debug (though the API spec supports it, the CLI does not expose this).
  • Resource limits. Ephemeral containers do not support resource requests and limits in the kubectl debug CLI, though this is evolving.

Accessing the Application Filesystem

The most common surprise for developers new to ephemeral containers is that they cannot directly browse the application container’s filesystem. The workaround is the /proc filesystem:

# Find the application's PID
ps aux

# Browse its filesystem via /proc
ls /proc/1/root/app/
cat /proc/1/root/etc/config.yaml

# Or set the root to the application's root
chroot /proc/1/root /bin/sh  # only if /bin/sh exists in the app image

The /proc/<pid>/root path is a symlink to the container’s root filesystem as seen from the process namespace. Because the ephemeral container shares the process namespace with --target, the application’s PID is typically 1, and /proc/1/root gives you full read access to its filesystem.

RBAC Requirements

Ephemeral containers require the pods/ephemeralcontainers subresource permission. This is separate from pods/exec, which controls kubectl exec. A common mistake is to grant pods/exec for debugging purposes without realizing that ephemeral containers require an additional grant:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ephemeral-debugger
rules:
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

In production environments, this permission should be tightly scoped: time-limited via RoleBinding rather than permanent ClusterRoleBinding, restricted to specific namespaces, and ideally gated behind an approval workflow. The debug container runs as root by default, which can create privilege escalation paths if the application container runs as a non-root user with shared process namespace — the debug container can attach to the application’s processes with higher privileges.

Option 2: kubectl debug –copy-to (Pod Copy Strategy)

When you need to modify the pod’s container spec — replace the image, change environment variables, add a sidecar with a shared filesystem — the --copy-to flag creates a full copy of the pod with your modifications applied:

kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=my-app:debug \
  --share-processes

This creates a new pod named my-pod-debug that is a copy of my-pod but with the container image replaced by my-app:debug. If my-app:debug is your application image built with debug tooling included (or a debug variant from your registry), this lets you interact with the exact same binary in the exact same configuration as the original pod.

A more common use of --copy-to is to attach a debug container alongside the existing application container while keeping the original image unchanged:

kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  --container=debugger

This creates the copy-pod with both the original containers and a new debugger container sharing the process namespace. Unlike ephemeral containers, this approach supports volume mounts and resource limits, and the debug pod can be deleted cleanly when you are done.

Limitations of the Copy Strategy

The pod copy approach has a critical limitation: it is not debugging the original pod. It creates a new pod that may behave differently because:

  • It does not share the original pod’s in-memory state — if the issue is a goroutine leak or heap corruption that has been accumulating for hours, the fresh copy will not exhibit it immediately
  • It creates a new Pod UID, which means any admission webhooks, network policies, or pod-level security contexts that depend on pod identity may apply differently
  • If the original pod is crashing (CrashLoopBackOff), the copy will also crash — this technique does not help for crash debugging unless you also change the entrypoint

For crash debugging specifically, combine --copy-to with a modified entrypoint to keep the container alive:

kubectl debug my-crashing-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  -- sleep 3600

Option 3: Debug Image Variants

The most pragmatic approach — and the one most appropriate for developer workflows — is to maintain a debug variant of your application image that includes shell tooling. Both the Google distroless project and Chainguard provide this pattern officially.

Google distroless images have a :debug tag that adds BusyBox to the image:

# Production image
FROM gcr.io/distroless/java17-debian12

# Debug variant — identical but with BusyBox shell
FROM gcr.io/distroless/java17-debian12:debug

Chainguard images follow a similar convention with :latest-dev variants that include apk, a shell, and common utilities:

# Production (zero shell, minimal footprint)
FROM cgr.dev/chainguard/go:latest

# Development/debug variant
FROM cgr.dev/chainguard/go:latest-dev

If you build your own base images, the recommended approach is to use multi-stage builds and maintain separate build targets:

FROM golang:1.22 AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp .

# Production: static distroless image
FROM gcr.io/distroless/static-debian12 AS production
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

# Debug variant: same binary, with shell tools
FROM gcr.io/distroless/static-debian12:debug AS debug
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

In your CI/CD pipeline, build both targets and push my-app:${VERSION} (production) and my-app:${VERSION}-debug (debug variant) to your registry. The debug image is never deployed to production by default, but it exists and is ready to be used with kubectl debug --copy-to when needed.

Security Considerations for Debug Variants

Debug image variants defeat much of the security benefit of distroless if they are used in production, even temporarily. Track usage carefully: log when debug images are deployed, require explicit approval, and ensure they are removed after the debugging session. In regulated environments, consider whether deploying a debug variant to production namespaces is permitted by your security policy — in many cases it is not, and you must use ephemeral containers (which add a debug process to the pod without modifying the application image) instead.

Option 4: cdebug

cdebug is an open-source CLI tool that simplifies distroless debugging by wrapping kubectl debug with more ergonomic defaults and additional capabilities. Its primary value is in making ephemeral container debugging feel like a native shell experience:

# Install
brew install cdebug
# or: go install github.com/iximiuz/cdebug@latest

# Debug a running pod
cdebug exec -it my-pod

# Specify a namespace and container
cdebug exec -it -n production my-pod -c my-container

# Use a specific debug image
cdebug exec -it my-pod --image=nicolaka/netshoot

What cdebug adds over raw kubectl debug:

  • Automatic filesystem chroot. cdebug exec automatically sets the filesystem root of the debug container to the target container’s filesystem, so you browse / and see the application’s files — not the debug image’s files. This addresses the most common friction point with kubectl debug.
  • Docker integration. cdebug exec works identically for Docker containers (cdebug exec -it <container-id>), making it the same muscle memory for local and cluster debugging.
  • No RBAC complications for Docker-based local development — useful for developer workflows before the code reaches Kubernetes.

The tradeoff: cdebug is a third-party dependency and requires installation. In environments with strict tooling policies (regulated industries, air-gapped clusters), it may not be an option. In those cases, the raw kubectl debug workflow with /proc/1/root filesystem navigation is the baseline.

Option 5: Node-Level Debugging

When everything else fails — the pod is in CrashLoopBackOff too fast to attach to, the issue is a kernel-level problem, or you need tools like strace that require elevated privileges — node-level debugging gives you direct access to the container’s processes from the host node.

kubectl debug node/ creates a privileged pod on the target node that mounts the node’s root filesystem under /host:

kubectl debug node/my-node-name \
  -it \
  --image=nicolaka/netshoot

From this privileged pod, you can use nsenter to enter the namespaces of any container running on the node:

# Find the container's PID on the node
# (from within the node debug pod)
crictl ps | grep my-container
crictl inspect <container-id> | grep pid

# Enter the container's namespaces
nsenter -t <pid> -m -u -i -n -p -- /bin/sh

# Or just the network namespace (for network debugging)
nsenter -t <pid> -n -- ip a

The nsenter approach lets you run tools from the node’s or debug container’s toolset while operating in the namespaces of the target container. This is how you run strace against a distroless process: strace is not in the application container, but you can run it from the node level while targeting the application’s PID.

# Trace all syscalls from the application process
nsenter -t <pid> -- strace -p <pid> -f -e trace=network

RBAC and Security for Node Debugging

Node-level debugging requires nodes/proxy and the ability to create privileged pods, which in most production clusters is restricted to cluster administrators. The debug pod runs with hostPID: true and hostNetwork: true, giving it visibility into all processes and network traffic on the node — not just the target container. This is significant: every process running on the node, including those in other tenants’ namespaces, is visible.

This technique should be treated as a break-glass procedure: log the access, require dual approval in production environments, and clean up immediately after the debugging session with kubectl delete pod --selector=app=node-debugger.

Choosing the Right Approach: Access Profile and Environment Matrix

The technique you should use depends on two axes: who you are (developer, platform engineer, ops/SRE) and where the issue is (local development, staging, production). The requirements and constraints differ significantly across these combinations.

Developer — Local or Development Cluster

Goal: Reproduce and understand a bug, inspect configuration, verify network connectivity to services.
Constraints: None material — full cluster admin on local or personal dev namespace.
Recommended approach: Debug image variants or cdebug.

In local development (Minikube, Kind, Docker Desktop), the fastest path is to build the debug variant of your image and deploy it directly. If you are working with another team’s service, cdebug exec gives you a shell in the container with automatic filesystem root without any special RBAC. The goal is speed and iteration — reserve the more structured approaches for higher environments.

Developer — Staging Cluster

Goal: Debug integration issues, inspect live configuration, verify environment-specific behavior.
Constraints: Shared cluster — cannot deploy arbitrary workloads to other teams’ namespaces, but has pods/ephemeralcontainers in own namespace.
Recommended approach: kubectl debug with ephemeral containers (--target), scoped to own namespace.

Staging is where ephemeral containers earn their keep. You can attach to a running pod without restarting it, without modifying the deployment spec, and without affecting other users of the same cluster. Grant developers pods/ephemeralcontainers in their team’s namespaces and they can self-service debug without needing ops involvement.

Platform Engineer / SRE — Production

Goal: Diagnose a live production incident. The pod is behaving unexpectedly — high latency, memory growth, unexpected connections, incorrect responses.
Constraints: Changes to running pods are high-risk. Any debug image deployment must be gated. The issue is live and affecting users.
Recommended approach: kubectl debug with ephemeral containers (ephemeral containers do not restart the pod, do not modify the deployment, and are auditable via API audit logs).

The key production requirements are auditability and minimal blast radius. Ephemeral containers satisfy both: they are recorded in the Kubernetes API audit log (who attached, when, to which pod), they do not modify the running application container, and they are limited to the pod’s own network and process namespaces. Document the debug session in your incident ticket: pod name, time, what was observed, who ran the debug container.

The --copy-to strategy is generally inappropriate for production incident response: it creates a new pod that may or may not exhibit the issue, it adds load to the cluster during an incident, and if it is attached to the same services (databases, downstream APIs), it produces additional traffic that complicates forensics.

Platform Engineer — Production, Node-Level Issue

Goal: Diagnose a kernel-level issue, a container runtime problem, a networking issue that spans multiple pods, or a situation where the pod is crashing too fast to attach to.
Constraints: Maximum privilege required. High operational risk.
Recommended approach: Node-level debug pod with nsenter. Treat as break-glass.

For this scenario, create a dedicated RBAC role that grants nodes/proxy access and the ability to create pods with hostPID: true in a dedicated debug namespace. Bind it only to specific users, require a separate authentication step (e.g., kubectl auth can-i check against a time-limited binding), and log all access. This level of access should generate a PagerDuty-style alert so that the security team knows a privileged debug session is active in production.

Common Errors and Solutions

Error: “ephemeral containers are disabled for this cluster”

Ephemeral containers require Kubernetes 1.16+ (alpha, behind feature gate) and are stable from 1.25. If you are on 1.16–1.22, you need to enable the EphemeralContainers feature gate on the API server and kubelet. From 1.23 it was beta and enabled by default. From 1.25 it is stable and always on. On managed Kubernetes services (EKS, GKE, AKS), check the cluster version — versions older than 1.25 may still have it disabled depending on your configuration.

Error: “cannot update ephemeralcontainers” (RBAC)

You have pods/exec but not pods/ephemeralcontainers. Add the grant shown in the RBAC section above. Note that pods/exec and pods/ephemeralcontainers are separate subresources — having one does not imply the other.

Error: “container not found” with –target

The container name in --target must match exactly the container name as defined in the Pod spec — not the image name. Check with kubectl get pod my-pod -o jsonpath='{.spec.containers[*].name}' to get the exact container names.

Error: Can see processes but cannot read /proc/1/root

The application container runs as a non-root user (e.g., UID 1000) and the ephemeral container runs as root. The application’s filesystem may have files owned by UID 1000 that are not readable by other UIDs depending on permissions. The /proc/<pid>/root path itself requires CAP_SYS_PTRACE capability. If your cluster’s PodSecurityStandards (PSS) are set to restricted, the debug container may not have this capability. Use the Baseline PSS profile for debug namespaces or explicitly add SYS_PTRACE to the ephemeral container’s securityContext.

Error: tcpdump shows no traffic

When using nicolaka/netshoot for network debugging, ensure the ephemeral container is created without --target if your goal is to capture all traffic on the pod’s network interface (not just the specific container’s process). With --target, you share the process namespace but the network namespace is shared at the pod level regardless. Run tcpdump -i any to capture on all interfaces including loopback, which is where inter-container traffic within a pod travels.

Decision Framework

Use this as a starting point to select the right technique for your situation:

ScenarioTechniqueRequirement
Active production incident, pod runningkubectl debug + ephemeral containerpods/ephemeralcontainers RBAC, k8s 1.25+
Pod crashing too fast to attachkubectl debug –copy-to + modified entrypointAbility to create pods in namespace
Developer debugging in dev/stagingcdebug exec or kubectl debugpods/ephemeralcontainers or pod create
Need full filesystem accesskubectl debug –copy-to + debug image variantDebug image in registry, pod create
Need strace or kernel tracingNode-level debug with nsenternodes/proxy, cluster admin equivalent
Network packet capturekubectl debug + nicolaka/netshootpods/ephemeralcontainers
Local Docker debuggingcdebug exec <container-id>Docker socket access
CI-reproducible debug environmentDebug image variant in separate build targetSeparate image tag in registry

Production RBAC Design

A clean RBAC design for production distroless debugging separates three roles with different privilege levels:

# Tier 1: Developer self-service in team namespaces
# Allows attaching ephemeral containers, no node access
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: distroless-debugger
  namespace: team-namespace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
---
# Tier 2: SRE production incident access
# Ephemeral containers across all namespaces
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sre-distroless-debugger
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
---
# Tier 3: Break-glass node access
# Only for platform team, time-limited binding recommended
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-debugger
rules:
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create", "get", "list", "delete"]
  # Restrict to debug namespace via RoleBinding, not ClusterRoleBinding

Bind Tier 1 permanently to your developers. Bind Tier 2 to SREs permanently but with audit alerts on use. Bind Tier 3 only on-demand (via a Kubernetes operator that creates time-limited RoleBindings) and never as a permanent ClusterRoleBinding.

Summary

Distroless containers are the correct choice for production workloads. They reduce attack surface, eliminate unnecessary CVEs, and force a cleaner separation between application and tooling. The operational cost is that your traditional debugging workflow — exec into the container, run some commands — no longer works by default.

Kubernetes provides a clean answer with ephemeral containers and kubectl debug: inject a debug container with whatever tools you need into the running pod, sharing its network and process namespaces, without restarting or modifying the application. For scenarios where ephemeral containers are insufficient — filesystem access, crash debugging, kernel-level investigation — the copy strategy and node-level debug fill the remaining gaps.

The key to making this work at scale is not the technique itself but the access model: developers get self-service ephemeral container access in their own namespaces, SREs get cluster-wide ephemeral container access for production incidents, and node-level access is a break-glass procedure with audit trail and time limits. With that model in place, distroless becomes an operational non-issue rather than an obstacle.

Integrate Kyverno CLI into CI/CD Pipelines with GitHub Actions for Kubernetes Policy Checks

Integrate Kyverno CLI into CI/CD Pipelines with GitHub Actions for Kubernetes Policy Checks

Introduction

As Kubernetes clusters become an integral part of infrastructure, maintaining compliance with security and configuration policies is crucial. Kyverno, a policy engine designed for Kubernetes, can be integrated into your CI/CD pipelines to enforce configuration standards and automate policy checks. In this article, we’ll walk through integrating Kyverno CLI with GitHub Actions, providing a seamless workflow for validating Kubernetes manifests before they reach your cluster.

What is Kyverno CLI?

Kyverno is a Kubernetes-native policy management tool, enabling users to enforce best practices, security protocols, and compliance across clusters. Kyverno CLI is a command-line interface that lets you apply, test, and validate policies against YAML manifests locally or in CI/CD pipelines. By integrating Kyverno CLI with GitHub Actions, you can automate these policy checks, ensuring code quality and compliance before deploying resources to Kubernetes.

Benefits of Using Kyverno CLI in CI/CD Pipelines

Integrating Kyverno into your CI/CD workflow provides several advantages:

  1. Automated Policy Validation: Detect policy violations early in the CI/CD pipeline, preventing misconfigured resources from deployment.
  2. Enhanced Security Compliance: Kyverno enables checks for security best practices and compliance frameworks.
  3. Faster Development: Early feedback on policy violations streamlines the process, allowing developers to fix issues promptly.

Setting Up Kyverno CLI in GitHub Actions

Step 1: Install Kyverno CLI

To use Kyverno in your pipeline, you need to install the Kyverno CLI in your GitHub Actions workflow. You can specify the Kyverno version required for your project or use the latest version.

Here’s a sample GitHub Actions YAML configuration to install Kyverno CLI:

name: CI Pipeline with Kyverno Policy Checks

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  kyverno-policy-check:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v2

      - name: Install Kyverno CLI
        run: |
          curl -LO https://github.com/kyverno/kyverno/releases/download/v<version>/kyverno-cli-linux.tar.gz
          tar -xzf kyverno-cli-linux.tar.gz
          sudo mv kyverno /usr/local/bin/

Replace <version> with the version of Kyverno CLI you wish to use. Alternatively, you can replace it with latest to always fetch the latest release.

Step 2: Define Policies for Validation

Create a directory in your repository to store Kyverno policies. These policies define the standards that your Kubernetes resources should comply with. For example, create a directory structure as follows:

.
└── .github
    └── policies
        ├── disallow-latest-tag.yaml
        └── require-requests-limits.yaml

Each policy is defined in YAML format and can be customized to meet specific requirements. Below are examples of policies that might be used:

  • Disallow latest Tag in Images: Prevents the use of the latest tag to ensure version consistency.
  • Enforce CPU/Memory Limits: Ensures resource limits are set for containers, which can prevent resource abuse.

Step 3: Add a GitHub Actions Step to Validate Manifests

In this step, you’ll use Kyverno CLI to validate Kubernetes manifests against the policies defined in the .github/policies directory. If a manifest fails validation, the pipeline will halt, preventing non-compliant resources from being deployed.

Here’s the YAML configuration to validate manifests:

- name: Validate Kubernetes Manifests
  run: |
    kyverno apply .github/policies -r manifests/

Replace manifests/ with the path to your Kubernetes manifests in the repository. This command applies all policies in .github/policies against each YAML file in the manifests directory, stopping the pipeline if any non-compliant configurations are detected.

Step 4: Handle Validation Results

To make the output of Kyverno CLI more readable, you can use additional GitHub Actions steps to format and handle the results. For instance, you might set up a conditional step to notify the team if any manifest is non-compliant:

- name: Check for Policy Violations
  if: failure()
  run: echo "Policy violation detected. Please review the failed validation."

Alternatively, you could configure notifications to alert your team through Slack, email, or other integrations whenever a policy violation is identified.

Example: Validating a Kubernetes Manifest

Suppose you have a manifest defining a Kubernetes deployment as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest  # Should trigger a violation

The policy disallow-latest-tag.yaml checks if any container image uses the latest tag and rejects it. When this manifest is processed, Kyverno CLI flags the image and halts the CI/CD pipeline with an error, preventing the deployment of this manifest until corrected.

Conclusion

Integrating Kyverno CLI into a GitHub Actions CI/CD pipeline offers a robust, automated solution for enforcing Kubernetes policies. With this setup, you can ensure Kubernetes resources are compliant with best practices and security standards before they reach production, enhancing the stability and security of your deployments.

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

Kyverno: A Detailed Way of Enforcing Standard and Custom Policies

Kyverno: A Detailed Way of Enforcing Standard and Custom Policies

In the Kubernetes ecosystem, security and governance are key aspects that need continuous attention. While Kubernetes offers some out-of-the-box (OOTB) security features such as Pod Security Admission (PSA), these might not be sufficient for complex environments with varying compliance requirements. This is where Kyverno comes into play, providing a powerful yet flexible solution for managing and enforcing policies across your cluster.

In this post, we will explore the key differences between Kyverno and PSA, explain how Kyverno can be used in different use cases, and show you how to install and deploy policies with it. Although custom policy creation will be covered in a separate post, we will reference some pre-built policies you can use right away.

What Is Kyverno?

Kyverno is a policy engine for Kubernetes that lets you validate, mutate and generate cluster resources using policies written as plain YAML — no new programming language required. It runs as an admission controller: when someone applies a resource, Kyverno intercepts the request and either allows it, rejects it, silently fixes it, or creates additional resources alongside it, according to the rules you have defined.

The name means “govern” in Greek, and that is a fair description of the job. Typical uses are enforcing that every pod sets resource limits, blocking images from untrusted registries, requiring specific labels on namespaces, automatically injecting a NetworkPolicy into every new namespace, or verifying image signatures before a workload is admitted.

Three properties are worth knowing before you evaluate it:

  • Policies are Kubernetes resources. A Kyverno policy is a ClusterPolicy or Policy object written in YAML, so it goes through the same review, GitOps and RBAC machinery as the rest of your manifests.
  • It does more than say no. Validation is only one of three modes. Kyverno can also mutate (patch a resource on the way in) and generate (create related resources automatically), which is where most of its day-to-day value ends up coming from.
  • It is Kubernetes-only by design. That is a deliberate trade-off, and it is the main axis on which it differs from OPA Gatekeeper — see the comparison below.

The current release at the time of writing is Kyverno v1.18.2 (July 2026), and recent versions support both YAML-based and CEL-based policy expressions.

Kyverno vs OPA Gatekeeper: Which Policy Engine?

This is the comparison most teams actually need to make, and after several years of both projects converging it is rarely about raw capability. The short version: choose Kyverno if your policies are Kubernetes-only and you want them in YAML; choose OPA Gatekeeper if you need maximum expressiveness or want to reuse the same policy language outside Kubernetes.

KyvernoOPA Gatekeeper
Policy languageYAML (plus CEL); nothing new to learnRego, via ConstraintTemplates and Constraints
Learning curveLow — it looks like the manifests you already writeHigher — Rego is a genuine language to learn
Validate✅ first-class✅ first-class, its strongest area
Mutate✅ first-class policy type✅ added later, via dedicated resources rather than Rego
Generate resources✅ native❌ not a design goal
ScopeKubernetes onlyKubernetes, plus APIs, microservices, Terraform — same Rego everywhere
Complex programmatic logicWorkable, but YAML shows its limitsWhere Rego pulls ahead
CNCF maturityGraduatedGraduated (part of Open Policy Agent)
Current releasev1.18.2 (Jul 2026)v3.23.0 (Jul 2026)

There is now a third option that did not exist when this debate started: ValidatingAdmissionPolicy with CEL, built into Kubernetes itself. It needs no extra components at all, so if a rule can be expressed in CEL it is worth reaching for first. The limitation is that CEL cannot make external calls or hold state — so image-signature verification, cross-resource lookups, mutation and generation still belong to Kyverno or Gatekeeper. In practice many clusters end up with native CEL policies for the simple rules and Kyverno for everything else.

Who Maintains Kyverno? Governance and Commercial Support

Kyverno is a CNCF Graduated project — the same maturity tier as Kubernetes, Prometheus and Envoy, and the highest the foundation awards. That matters for the question every platform team eventually asks: the project is not owned by a single vendor who can change the licence, and graduation requires a demonstrated governance model, security audit and a healthy contributor base.

It was created by Nirmata, who remain the most active contributor and offer commercial support and long-term support around it. If you need a support contract, an LTS commitment or consulting, Nirmata is the primary vendor, and several others also provide commercial Kyverno support — Giant Swarm, InfraCloud, BlakYaks and Kodekloud among them. The open-source project itself is free and Apache-2.0, and nothing in the upstream distribution is gated behind a commercial tier.

What is Pod Security Admission (PSA)?

Kubernetes introduced Pod Security Admission (PSA) as a replacement for the now deprecated PodSecurityPolicy (PSP). PSA focuses on enforcing three predefined levels of security: Privileged, Baseline, and Restricted. These levels control what pods are allowed to run in a namespace based on their security context configurations.

  • Privileged: Minimal restrictions, allowing privileged containers and host access.
  • Baseline: Applies standard restrictions, disallowing privileged containers and limiting host access.
  • Restricted: The strictest level, ensuring secure defaults and enforcing best practices for running containers.

While PSA is effective for basic security requirements, it lacks flexibility when enforcing fine-grained or custom policies. We have a full article covering this topic that you can read here.

Kyverno vs. PSA: Key Differences

Kyverno extends beyond the capabilities of PSA by offering more granular control and flexibility. Here’s how it compares:

  1. Policy Types: While PSA focuses solely on security, Kyverno allows the creation of policies for validation, mutation, and generation of resources. This means you can modify or generate new resources, not just enforce security rules.
  2. Customizability: Kyverno supports custom policies that can enforce your organization’s compliance requirements. You can write policies that govern specific resource types, such as ensuring that all deployments have certain labels or that container images come from a trusted registry.
  3. Policy as Code: Kyverno policies are written in YAML, allowing for easy integration with CI/CD pipelines and GitOps workflows. This makes policy management declarative and version-controlled, which is not the case with PSA.
  4. Audit and Reporting: With Kyverno, you can generate detailed audit logs and reports on policy violations, giving administrators a clear view of how policies are enforced and where violations occur. PSA lacks this built-in reporting capability.
  5. Enforcement and Mutation: While PSA primarily enforces restrictions on pods, Kyverno allows not only validation of configurations but also modification of resources (mutation) when required. This adds an additional layer of flexibility, such as automatically adding annotations or labels.

When to Use Kyverno Over PSA

While PSA might be sufficient for simpler environments, Kyverno becomes a valuable tool in scenarios requiring:

  • Custom Compliance Rules: For example, enforcing that all containers use a specific base image or restricting specific container capabilities across different environments.
  • CI/CD Integrations: Kyverno can integrate into your CI/CD pipelines, ensuring that resources comply with organizational policies before they are deployed.
  • Complex Governance: When managing large clusters with multiple teams, Kyverno’s policy hierarchy and scope allow for finer control over who can deploy what and how resources are configured.

If your organization needs a more robust and flexible security solution, Kyverno is a better fit compared to PSA’s more generic approach.

Installing Kyverno

To start using Kyverno, you’ll need to install it in your Kubernetes cluster. This is a straightforward process using Helm, which makes it easy to manage and update.

Step-by-Step Installation

Add the Kyverno Helm repository:

    helm repo add kyverno https://kyverno.github.io/kyverno/

    Update Helm repositories:

      helm repo update

      Install Kyverno in your Kubernetes cluster:

        helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace

        Verify the installation:

          kubectl get pods -n kyverno

          After installation, Kyverno will begin enforcing policies across your cluster, but you’ll need to deploy some policies to get started.

          Installing the Kyverno CLI

          Separate from the in-cluster controller, Kyverno ships a CLI (kyverno) used to test and validate policies before they reach a cluster. It is the piece that makes policies reviewable in a pull request rather than discovered in production.

          # macOS / Linux (Homebrew)
          brew install kyverno
          
          # Krew (as a kubectl plugin)
          kubectl krew install kyverno
          
          # Direct binary
          curl -LO https://github.com/kyverno/kyverno/releases/latest/download/kyverno-cli_linux_x86_64.tar.gz
          tar -xvf kyverno-cli_linux_x86_64.tar.gz && sudo mv kyverno /usr/local/bin/

          The two commands worth knowing immediately:

          # Apply a policy against manifests without touching a cluster
          kyverno apply ./policies/ --resource ./manifests/
          
          # Run the policy test suite defined in kyverno-test.yaml
          kyverno test ./policies/

          kyverno apply answers “would this policy have blocked this manifest?” and prints a policy report; kyverno test runs declarative test cases so policies have regression coverage like any other code. Wiring both into CI is covered step by step in integrating the Kyverno CLI into CI/CD pipelines with GitHub Actions.

          The Kyverno Policy Library

          Before writing anything yourself, check the official Kyverno policy library at kyverno.io/policies. It carries several hundred ready-made policies grouped by category — Pod Security Standards equivalents, best-practice rules, multi-tenancy controls, image verification, and cleanup policies. Most teams find that their first ten policies already exist there, and the realistic starting point is to adopt the Pod Security Standards set in audit mode, see what your cluster is already violating, and only then start writing custom rules.

          Deploying Policies with Kyverno

          Kyverno policies are written in YAML, just like Kubernetes resources, which makes them easy to read and manage. You can find several ready-to-use policies from the Kyverno Policy Library, or create your own to match your requirements.

          Here is an example of a simple validation policy that ensures all pods use trusted container images from a specific registry:

          apiVersion: kyverno.io/v1
          kind: ClusterPolicy
          metadata:
            name: require-trusted-registry
          spec:
            validationFailureAction: enforce
            rules:
            - name: check-registry
              match:
                resources:
                  kinds:
                  - Pod
              validate:
                message: "Only images from 'myregistry.com' are allowed."
                pattern:
                  spec:
                    containers:
                    - image: "myregistry.com/*"

          This policy will automatically block the deployment of any pod that uses an image from a registry other than myregistry.com.

          Applying the Policy

          To apply the above policy, save it to a YAML file (e.g., trusted-registry-policy.yaml) and run the following command:

          kubectl apply -f trusted-registry-policy.yaml

          Once applied, Kyverno will enforce this policy across your cluster.

          Viewing Kyverno Policy Reports

          Kyverno generates detailed reports on policy violations, which are useful for audits and tracking policy compliance. To check the reports, you can use the following commands:

          List all Kyverno policy reports:

            kubectl get clusterpolicyreport

            Describe a specific policy report to get more details:

              kubectl describe clusterpolicyreport <report-name>

              These reports can be integrated into your monitoring tools to trigger alerts when critical violations occur.

              Frequently Asked Questions

              What is Kyverno used for?

              Kyverno is a Kubernetes policy engine used to validate, mutate and generate cluster resources. Typical uses are enforcing that pods declare resource limits, blocking images from untrusted registries, requiring labels on namespaces, automatically generating a NetworkPolicy or ConfigMap in every new namespace, and verifying image signatures before a workload is admitted. Policies are written as YAML and applied as Kubernetes objects.

              Kyverno vs OPA Gatekeeper: which should I use?

              Use Kyverno if your policies only target Kubernetes and you want them written in YAML with no new language to learn — it also handles mutation and resource generation natively. Use OPA Gatekeeper if you need highly expressive or programmatic policy logic, or if you want to reuse the same Rego policies outside Kubernetes (APIs, microservices, Terraform). Both are CNCF Graduated and have converged on feature parity for core admission control, so the decision is usually about language and scope rather than capability.

              Is Kyverno free? Who is the company behind it?

              The Kyverno project is free and open source under the Apache-2.0 licence, and it is a CNCF Graduated project, so it is not controlled by a single vendor. It was created by Nirmata, who remain the largest contributor and offer commercial support and long-term support. Several other vendors also provide commercial Kyverno support, including Giant Swarm, InfraCloud, BlakYaks and Kodekloud.

              How do I install the Kyverno CLI?

              The quickest routes are brew install kyverno on macOS or Linux, or kubectl krew install kyverno to use it as a kubectl plugin. Binaries for each platform are also published on the GitHub releases page. The CLI is separate from the in-cluster controller: it is used to run kyverno apply and kyverno test against manifests in CI, before policies ever reach a cluster.

              Do I still need Kyverno now that Kubernetes has ValidatingAdmissionPolicy?

              For simple rules, often not — ValidatingAdmissionPolicy with CEL is built into Kubernetes, needs no extra components, and should be your first choice when the rule can be expressed in CEL. But CEL cannot make external calls or maintain state, and it only validates. Image-signature verification, cross-resource lookups, mutating resources on admission and generating new resources all still require Kyverno or Gatekeeper. Many clusters run both.

              What is the difference between Kyverno and Pod Security Admission?

              Pod Security Admission is built into Kubernetes and enforces the three fixed Pod Security Standards profiles (privileged, baseline, restricted) at namespace level. It is simple but not extensible — you cannot add a rule of your own. Kyverno is a general policy engine: it can reproduce the PSS profiles and then go further with custom rules, mutation and generation, and it applies to any resource type rather than pods alone.

              Conclusion

              Kyverno offers a flexible and powerful way to enforce policies in Kubernetes, making it an essential tool for organizations that need more than the basic capabilities provided by PSA. Whether you need to ensure compliance with internal security standards, automate resource modifications, or integrate policies into CI/CD pipelines, Kyverno’s extensive feature set makes it a go-to choice for Kubernetes governance.

              For now, start with the out-of-the-box policies available in Kyverno’s library. In future posts, we’ll dive deeper into creating custom policies tailored to your specific needs.

              📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

              Kubernetes Pod Security Admission Explained: Enforcing PSA Policies the Right Way

              Kubernetes Pod Security Admission Explained: Enforcing PSA Policies the Right Way

              In Kubernetes, security is a key concern, especially as containers and microservices grow in complexity. One of the essential features of Kubernetes for policy enforcement is Pod Security Admission (PSA), which replaces the deprecated Pod Security Policies (PSP). PSA provides a more straightforward and flexible approach to enforce security policies, helping administrators safeguard clusters by ensuring that only compliant pods are allowed to run.

              This article will guide you through PSA, the available Pod Security Standards, how to configure them, and how to apply security policies to specific namespaces using labels.

              What is Pod Security Admission (PSA)?

              PSA is a built-in admission controller introduced in Kubernetes 1.23 to replace Pod Security Policies (PSPs). PSPs had a steep learning curve and could become cumbersome when scaling security policies across various environments. PSA simplifies this process by applying Kubernetes Pod Security Standards based on predefined security levels without needing custom logic for each policy.

              With PSA, cluster administrators can restrict the permissions of pods by using labels that correspond to specific Pod Security Standards. PSA operates at the namespace level, enabling better granularity in controlling security policies for different workloads.

              Pod Security Standards

              Kubernetes provides three key Pod Security Standards in the PSA framework:

              • Privileged: No restrictions; permits all features and is the least restrictive mode. This is not recommended for production workloads but can be used in controlled environments or for workloads requiring elevated permissions.
              • Baseline: Provides a good balance between usability and security, restricting the most dangerous aspects of pod privileges while allowing common configurations. It is suitable for most applications that don’t need special permissions.
              • Restricted: The most stringent level of security. This level is intended for workloads that require the highest level of isolation and control, such as multi-tenant clusters or workloads exposed to the internet.

              Each standard includes specific rules to limit pod privileges, such as disallowing privileged containers, restricting access to the host network, and preventing changes to certain security contexts.

              Setting Up Pod Security Admission (PSA)

              To enable PSA, you need to label your namespaces based on the security level you want to enforce. The label format is as follows:

              kubectl label --overwrite ns  pod-security.kubernetes.io/enforce=<value>

              For example, to enforce a restricted security policy on the production namespace, you would run:

              kubectl label --overwrite ns production pod-security.kubernetes.io/enforce=restricted

              In this example, Kubernetes will automatically apply the rules associated with the restricted policy to all pods deployed in the production namespace.

              Additional PSA Modes

              PSA also provides additional modes for greater control:

              • Audit: Logs a policy violation but allows the pod to be created.
              • Warn: Issues a warning but permits the pod creation.
              • Enforce: Blocks pod creation if it violates the policy.

              To configure these modes, use the following labels:

              kubectl label --overwrite ns      pod-security.kubernetes.io/enforce=baseline     pod-security.kubernetes.io/audit=restricted     pod-security.kubernetes.io/warn=baseline

              This setup enforces the baseline standard while issuing warnings and logging violations for restricted-level rules.

              Example: Configuring Pod Security in a Namespace

              Let’s walk through an example of configuring baseline security for the dev namespace. First, you need to apply the PSA labels:

              kubectl create namespace dev
              kubectl label --overwrite ns dev pod-security.kubernetes.io/enforce=baseline

              Now, any pod deployed in the dev namespace will be checked against the baseline security standard. If a pod violates the baseline policy (for instance, by attempting to run a privileged container), it will be blocked from starting.

              You can also combine warn and audit modes to track violations without blocking pods:

              kubectl label --overwrite ns dev     pod-security.kubernetes.io/enforce=baseline     pod-security.kubernetes.io/warn=restricted     pod-security.kubernetes.io/audit=privileged

              In this case, PSA will allow pods to run if they meet the baseline policy, but it will issue warnings for restricted-level violations and log any privileged-level violations.

              Applying Policies by Default

              One of the strengths of PSA is its simplicity in applying policies at the namespace level, but administrators might wonder if there’s a way to apply a default policy across new namespaces automatically. As of now, Kubernetes does not natively provide an option to apply PSA policies globally by default. However, you can use admission webhooks or automation tools such as OPA Gatekeeper or Kyverno to enforce default policies for new namespaces.

              Conclusion

              Pod Security Admission (PSA) simplifies policy enforcement in Kubernetes clusters, making it easier to ensure compliance with security standards across different environments. By configuring Pod Security Standards at the namespace level and using labels, administrators can control the security level of workloads with ease. The flexibility of PSA allows for efficient security management without the complexity associated with the older Pod Security Policies (PSPs).

              For more details on configuring PSA and Pod Security Standards, check the official Kubernetes PSA documentation and Pod Security Standards documentation.

              📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

              KubeSec Explained: How to Scan and Improve Kubernetes Security with YAML Analysis

              KubeSec Explained: How to Scan and Improve Kubernetes Security with YAML Analysis

              KubeSec is another tool to help improve the security of our Kubernetes cluster. And we’re seeing so many agencies focus on security to highlight this topic’s importance in modern architectures and deployments. Security is a key component now, probably the most crucial. We need all to step up our game on that topic, and that’s why it is essential to have tools in our toolset to help us on that task without being fully security experts on each of the technologies, such as Kubernetes in this case.

              KubeSec is an open-source tool developed by a cloud-native and open-source security consultancy named ControlPlane that helps us perform a security risk analysis on Kubernetes resources.

              How Does KubeSec Work?

              KubeSec works based on the Kubernetes Manifest Files you use to deploy the different resources, so you need to provide the YAML file to one of the running ways this tool supports. This is an important topic, “one of the running ways,” because KubeSec supports many different running modes that help us cover other use cases.

              You can run KubeSec in the following ones:

              • HTTP Mode: KubeSec will be listening to HTTP requests with the content of the YAML and provide a report based on that. This is useful in cases needing server mode execution, such as CICD pipelines, or just security servers to be used by some teams, such as DevOps or Platform Engineering. Also, another critical use-case of this mode is to be part of a Kubernetes Admission Controller on your Kubernetes Cluster so that you can enforce this when developers are deploying resources into the platform itself.
              • SaaS Mode: Similar to HTTP mode but without needing to host it yourself, all available behind kubesec.io kubesec.io when the SaaS mode is of your preference, and you’re not managing sensitive information on those components.
              • CLI Mode: Just to run it yourself as part of your local tests, you will have available another CLI command here: kubesec scan k8s-deployment.yaml
              • Docker Mode: Similar to CLI mode but as part of a docker image, it can also be compatible with the CICD pipelines based on containerized workloads.

              KubeScan Output Report

              What you will get out of the execution if KubeScan of any of its forms is a JSON report that you can use to improve and score the security level of your Kubernetes resources and some ways to improve it. The reason behind using JSON as the output also simplifies the tool’s usage in automated workloads such as CICD pipelines. Here you can see a sample of the output report you will get:

              kubesec sample output

              The important thing about the output is the kind of information you will receive from it. As you can see in the picture above, it is separated into two different sections per object. The first one is the “score,” that are the implemented things related to security that provide some score for the security of the object. But also you will have an advice section that provides some things and configurations you can do to improve that score, and because of that, also the global security of the Kubernetes object itself.

              Kubescan also leverages another tool we have commented not far enough on this site, Kubeconform, so you can also specify the target Kubernetes version you’re hitting to have a much more precise report of your specific Kubernetes Manifest. To do that, you can specify the argument --kubernetes-version when you’re launching the command, as you can see in the picture below:

              kubesec command with kubernetes-version option

               How To Install KubeScan?

              Installation also provides different ways and flavors to see what is best for you. Here are some of the options available at the moment for writing this article:

              Conclusion

              Emphasizing the paramount importance of security in today’s intricate architectures, KubeSec emerges as a vital asset for bolstering the protection of Kubernetes clusters. Developed by ControlPlane, this open-source tool facilitates comprehensive security risk assessments of Kubernetes resources. Offering versatility through multiple operational modes—such as HTTP, SaaS, CLI, and Docker—KubeSec provides tailored support for diverse scenarios. Its JSON-based output streamlines integration into automated workflows, while its synergy with Kubeconform ensures precise analysis of Kubernetes Manifests. KubeSec’s user-friendly approach empowers security experts and novices, catalyzing an elevated standard of Kubernetes security across the board.

              📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

              Enable SwaggerUI in TIBCO BusinessWorks When Offloading SSL (BWCE Fix)

              Enable SwaggerUI in TIBCO BusinessWorks When Offloading SSL (BWCE Fix)

              SwaggerUI TIBCO BusinessWorks is one of the features available by default to all the TIBCO BusinessWorks REST Service developed. As you probably know, SwaggerUI is just an HTML Page with a graphical representation of the Swagger definition file (or OpenAPI specification to be more accurate with the current version of the standards in use) that helps to understand the operation and capabilities exposed by the service and also provide an easy way to test the service as you can see in the picture below:

              This article is part of my comprehensive TIBCO Integration Platform Guide where you can find more patterns and best practices for TIBCO integration platforms.

              How To Enable SwaggerUI TIBCO BusinessWorks when Offloading SSL Certificate: SwaggerUI view from TIBCO BWCE app

              This interface is provided out of the box for any REST Service developed using TIBCO BusinessWorks that uses a different port (7777 by default) in case we’re talking about an on-premises deployment or in the /swagger endpoint in case we are talking about a TIBCO BusinessWorks Container Edition.

               How does SwaggerUI work to load the Swagger Specification?

              SwaggerUI works in a particular way. When you reach the URL of the SwaggerUI, there is another URL that is usually part of a text field inside the web page that holds the link to the JSON or YAML document that stores the actual specification, as you can see in the picture below:

              How To Enable SwaggerUI TIBCO BusinessWorks when Offloading SSL Certificate: SwaggerUI highlighting the 2 URL loaded in the process

              So, you can think that this is a 2-call kind of process:

              • First call loads the SwaggerUI as a graphical container
              • Then, based on the internal URL provided there, do a second call to retrieve the document specification
              • And with that information, render the information in the SwaggerUI format.

              The issue is raised when the SwaggerUI is exposed behind a Load Balancer because the second URL needs to use the advertised URL as the backend server is not reached directly by the client browsing the SwaggerUI. This is solved out of the box with Kubernetes capabilities in the case of TIBCO BWCE, and for the on-premises deployment, it offers two properties to handle that as follows:

              # ------------------------------------------------------------------------------
              # Section:  BW REST Swagger Configuration.  The properties in this section
              # are applicable to the Swagger framework that is utilized by the BW REST 
              # Binding.
              #
              # Note: There are additional BW REST Swagger configuration properties that
              # can be specified in the BW AppNode configuration file "config.ini".  Refer to
              # the BW AppNode configuration file's section "BW REST Swagger configuration" 
              # for details. 
              # ------------------------------------------------------------------------------
              # Swagger framework reverse proxy host name.  This property is optional and 
              # it specifies the reverse proxy host name on which Swagger framework serves 
              # the API's, documentation  endpoint, api-docs, etc.. 
              bw.rest.docApi.reverseProxy.hostName=localhost
              
              # Swagger framework port.  This property is optional and it specifies the 
              # reverse proxy port on which Swagger framework serves the API's, documentation
              # endpoint, api-docs, etc.
              bw.rest.docApi.reverseProxy.port=0000
              

              You can browse the official documentation page for more detailed information.

              That solves the main issue regarding the hostname and the port to be reached as the final user requires. Still, there is an outstanding component on the URL that could generate an issue, and that’s the protocol, so, in a nutshell, if this is exposed using HTTP or HTTPS.

              How to Handle Swagger URL when offloading SSL?

              Until the release of TIBCO BWCE 2.8.3, the protocol depended on the HTTP Connector configuration you used to expose the swagger component. So, if you use an HTTP connector without SSL configuration, it will try to reach the endpoint using an HTTP connection. In the other case, if you use an HTTP connector with an SSL connection, it will try to use an HTTPS connection. That seems fine, but some use cases could generate a problem:

              SSL Certificate offloaded in the Load Balancer: If we offload the SSL configuration on the Load Balancer as it is used in traditional on-premises deployments and some of the Kubernetes configurations, the consumer will establish an HTTPS connection to the Load Balancer, but internally the communication with the BWCE will be done using HTTP, so, in this case, it will generate a mismatch, because in the second call of the requests it will guess that as the HTTP Connector from BWCE is not using HTTPS, the URL should be reached using HTTP but that’s not the case as the communication goes through the Load Balancer that is handled the security.

              Service Mesh Service Exposition: Similar to the previous case, but in that case, close to the Kubernetes deployment. Suppose we are using Service Mesh such as Istio or others. In that case, security is one of the things that needs to be handled. Hence, the situation is the same as the scenario above because the BWCE doesn’t know the security configuration but is impacting the default endpoint generated.

              How To Enable SwaggerUI TIBCO BusinessWorks when Offloading SSL Certificates?

              Since BWCE 2.8.3, there is a new JVM property that we can use to force the endpoint generated to be HTTPS even if the HTTP Connector used by the BWCE application doesn’t have any security configuration that helps us to solve this issue in the cases above and similar scenario. The property can be added as any other JVM property using the BW_JAVA_OPTS environment property, and the value is this: bw.rest.enable.secure.swagger.url =true

              ReadOnlyRootFilesystem for TIBCO BWCE: Securing Containers with Kubernetes Best Practices

              ReadOnlyRootFilesystem for TIBCO BWCE: Securing Containers with Kubernetes Best Practices

              This article will cover how to enhance the security of your TIBCO BWCE images by creating a ReadOnlyFileSystem Image for TIBCO BWCE. In previous articles, we have commented on the benefits that this kind of image provides several advantages in terms of security, focusing on aspects such as reducing the attack surface by limiting the kind of things any user can do, even if they gain access to running containers.

              This article is part of my comprehensive TIBCO Integration Platform Guide where you can find more patterns and best practices for TIBCO integration platforms.

              The same applies in case any malware your image can have will have limited the possible actions they can do without any write access to most of the container.

              How ReadOnlyFileSystem affects a TIBCO BWCE image?

              This has a clear impact as the TIBCO BWCE image is an image that needs to write in several folders as part of the expected behavior of the application. That’s mandatory and non-dependent on the scripts you used to build your image.

              As you probably know, TIBCO BWCE ships two sets of scripts to build the Docker base image: the main ones and the ones included in the folder reducedStartupTime, as you can see in the GitHub page but also inside your docker folder in the TIBCO-HOME after the installation as you can see in the picture below.

              ReadOnlyRootFilesystem for TIBCO BWCE: Securing Containers with Kubernetes Best Practices

              The main difference between them is where the unzip of the bwce-runtime is made. In the case of the default script, the unzip is done in the startup process of the image, and in the reducedStartupTime this is done in the building of the image itself. So, you can start thinking that the default ones need some writing access as they need to unzip the file inside the container, and that’s true.

              But also, the reduced startupTime requires writing access to run the application; several activities are done regarding unzipping the EAR file, managing the properties file, and additional internal activities. So, no matter what kind of scripts you’re using, you must provide a write-access folder to do this activity.

              By default, all these activities are limited to a single folder. If you keep everything by default, this is the /tmp folder, so you must provide a volume for that folder.

              How to deploy a TIBCO BWCE application with the

              Now, that is clear that you need a volume for the /tmp folder, and now you need to define the kind of volume that you want to use for this one. As you know, there are several kinds of volumes that you can determine depending on the requirements that you have.

              In this case, the only requirement is to write access, but there is no need regarding storage and persistency, so, in that case, we can use an emptyDir mode. emptyDir content, which is erased when a pod is removed, is similar to the default behavior but allows writing permission on its content.

              To show how the YAML would like, we will use the default one that we have available in the documentation here:

              apiVersion: v1
              kind: Pod
              metadata:
                name: bookstore-demo
                labels:
                  app: bookstore-demo
              spec:
                containers:
                - name: bookstore-demo
                  image: bookstore-demo:2.4.4
                  imagePullPolicy: Never
                  envFrom:
                  - configMapRef:
                    name: name 
              

              So, we will change that to include the volume, as you can see here:

              apiVersion: v1
              kind: Pod
              metadata:
                name: bookstore-demo
                labels:
                  app: bookstore-demo
              spec:
                containers:
                - name: bookstore-demo
                  image: bookstore-demo:2.4.4
                  imagePullPolicy: Never
              	securityContext:
              		readOnlyRootFilesystem: true
                  envFrom:
                  - configMapRef:
                    name: name
                  volumeMounts:
                  - name: tmp
                    mountPath: /tmp
                volumes:
                - name: tmp
                  emptyDir: {}
              

              The changes are the following:

              • Include the volumes section with a single volume definition with the name of tmp with an emptyDirdefinition.
              • Include a volumeMountssection for the tmpvolume that is mounted in the /tmp path to allow to write on that specific path to enable also the unzip of the bwce-runtime as well as all the additional activities that are required.
              • To trigger this behavior, include the readOnlyRootFilesystem flag in the securityContext section.

              Conclusion

              Incorporating a ReadOnlyFileSystem approach into your TIBCO BWCE images is a proactive strategy to fortify your application’s security posture. By curbing unnecessary write access and minimizing the potential avenues for unauthorized actions, you’re taking a vital step towards safeguarding your containerized environment.

              This guide has unveiled the critical aspects of implementing such a security-enhancing measure, walking you through the process with clear instructions and practical examples. With a focus on reducing attack vectors and bolstering isolation, you can confidently deploy your TIBCO BWCE applications, knowing that you’ve fortified their runtime environment against potential threats.