Kaniko, BuildKit, and Image Volumes: The Evolution of Container Images Inside Kubernetes

Kaniko, BuildKit, and Image Volumes: The Evolution of Container Images Inside Kubernetes

Running container images inside Kubernetes is table stakes. Building them there — or mounting their contents as volumes — has been a moving target for years. What started as a privileged hack has evolved into a set of mature, secure, and increasingly native primitives.

This article covers the full arc: why the original approach was broken, what Kaniko solved, why its maintenance status now matters, where BuildKit and Buildah fit, and what Image Volumes (stable in Kubernetes v1.36) change for workflows that don’t need to build anything at all.


The original problem: Docker-in-Docker

The first generation of CI/CD on Kubernetes ran Docker inside Docker. You mounted the host Docker socket (/var/run/docker.sock) into a build container, and that container had full access to the host’s Docker daemon.

# The approach nobody should be using in 2026
volumes:
- name: docker-sock
  hostPath:
    path: /var/run/docker.sock

This worked. It was also a complete security disaster.

Mounting the Docker socket gives the container root-equivalent access to the host. Any workload that can reach that socket can escape the container, inspect other containers, and compromise the node. The only thing standing between your CI pipeline and a full cluster compromise was the good intentions of whoever wrote the build script.

It got worse when the Kubernetes project deprecated Dockershim in v1.20 and removed it in v1.24. Clusters that moved to containerd or CRI-O no longer had a Docker daemon on the host at all. The socket either didn’t exist or belonged to a completely different runtime. Docker-in-Docker in its classic form became functionally impossible on modern clusters.


Kaniko: daemonless builds inside containers

Google released Kaniko in 2018 to solve exactly this problem. Kaniko builds container images entirely in userspace — no daemon, no privileged access, no host socket required.

The key insight: Docker builds work by executing each RUN instruction in a temporary container, snapshotting the filesystem, and saving the result as a layer. Kaniko replicates this logic without a daemon. It runs as a regular container, reads the Dockerfile, executes each step against the local filesystem, and pushes the resulting image directly to a registry.

That design aged well. The project governance did not. The GoogleContainerTools/kaniko repository was archived by its owner on June 3, 2025 and is now read-only. In practical terms, the original Google-hosted Kaniko project should be treated as unmaintained: no active upstream issue triage, no normal pull request flow, and no clear path for security fixes through that repository.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-build
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args:
    - "--dockerfile=Dockerfile"
    - "--context=git://github.com/your-org/your-repo"
    - "--destination=your-registry/your-image:tag"
    volumeMounts:
    - name: registry-creds
      mountPath: /kaniko/.docker
  volumes:
  - name: registry-creds
    secret:
      secretName: registry-credentials
      items:
      - key: .dockerconfigjson
        path: config.json
  restartPolicy: Never

No host socket. No privileged flag. The container needs write access to its own filesystem (so readOnlyRootFilesystem: true won’t work), but that’s a far narrower requirement than socket mounting.

When Kaniko is still a reasonable answer

Kaniko can still be a reasonable tactical choice when you need to build a container image inside Kubernetes and already have a working, isolated pipeline around it. Existing Kaniko jobs did not stop working when the repository was archived.

For new platform work in 2026, though, do not pick Kaniko by default. The maintenance signal changes the risk model. Use it only if the simplicity is worth owning the upgrade and vulnerability-management story yourself, or if you deliberately standardize on a maintained fork or vendor-supported distribution.

Kaniko integrates well with: – Tekton — the standard pattern is a Tekton Task running the Kaniko executor – Argo Workflows — same pattern, different orchestrator – GitLab CI on Kubernetes — many older examples and pipelines use Kaniko with the Kubernetes executor – Any pod-based CI system — it’s just a container, so it runs anywhere pods run


The alternatives: BuildKit and Buildah

Kaniko is no longer the only serious daemonless option. Two other tools are worth evaluating first for new work:

BuildKit (rootless mode)

BuildKit is the build backend behind docker buildx and the default build system in modern Docker. In rootless mode it runs without privileges and can build images inside a Kubernetes pod.

BuildKit has better caching than Kaniko — particularly layer caching via cache mounts — and supports more advanced Dockerfile features like heredocs and multi-platform builds. The tradeoff is more complex setup: you need to run buildkitd as a sidecar or as a DaemonSet.

For teams already using docker buildx locally, BuildKit is the most natural migration path. Docker’s Kubernetes driver can run BuildKit builders directly in a cluster, including rootless mode without privileged pods on supported Kubernetes versions. The operational cost is real — builder lifecycle, cache persistence, node placement, and rootless kernel requirements — but the project is active and the feature set is where most modern Dockerfile workflows are moving.

Buildah

Buildah is the containers project’s daemonless build tool, designed to integrate with Podman and OCI-native workflows. It can build from Dockerfiles or Containerfiles, and it is the natural choice on OpenShift or in environments where the platform team already standardizes on Red Hat, Podman, and the containers/* stack.

The caveat is that rootless Buildah inside a restricted Kubernetes pod still depends on user namespace behavior and helper binaries such as newuidmap and newgidmap. That is manageable on platforms built for it, especially OpenShift, but it is not automatically a drop-in replacement for every generic Kubernetes CI runner.


Image Volumes: a different problem entirely

Here is where the narrative splits. Everything above is about building images. Image Volumes solve a completely different problem: consuming the contents of an OCI image as a volume, without building anything.

The feature was introduced as alpha in Kubernetes v1.31, moved to beta in v1.33 with subPath and subPathExpr support, became beta enabled by default in v1.35, and graduated to stable (GA) in Kubernetes v1.36.

What it does

Image Volumes let you reference an OCI image in a pod’s volumes section and mount its filesystem contents directly into a container:

apiVersion: v1
kind: Pod
metadata:
  name: image-volume-example
spec:
  containers:
  - name: app
    image: debian
    command: ["sleep", "infinity"]
    volumeMounts:
    - name: config-data
      mountPath: /app/config
  volumes:
  - name: config-data
    image:
      reference: your-registry/your-config-image:v1.2.0
      pullPolicy: IfNotPresent

The container at /app/config sees the contents of your-config-image:v1.2.0. The volume is read-only. No init container required.

From Kubernetes v1.33+, you can also mount a subdirectory with subPath:

volumeMounts:
- name: config-data
  mountPath: /app/config
  subPath: environments/production

The use case: OCI images as artifact bundles

This feature is motivated by a pattern that has been growing quietly: using OCI images not as runnable containers but as versioned, signed, distributable artifact bundles.

The idea: instead of storing configuration, schemas, WASM modules, ML models, or static binaries in a ConfigMap or a separate volume, you package them as an OCI image. You get:

  • Version control — image tags and digests, same tooling you already use
  • Distribution — your existing registry, your existing pull secrets, your existing access controls
  • Signing and attestation — cosign, Sigstore, the full supply chain tooling works on these artifacts
  • Immutability — a digest-pinned image reference is cryptographically immutable

Image Volumes are the Kubernetes primitive that makes this pattern first-class. Without Image Volumes, the workaround was an init container that pulled the image and copied the contents to an emptyDir. It worked, but it was boilerplate.

What it does not do

Image Volumes are not a replacement for Kaniko or any build tool. They consume images; they don’t produce them. If your workflow involves building a new image from source, you still need Kaniko, BuildKit, or equivalent.

They also require container runtime support. CRI-O supported the initial alpha implementation from its v1.31 line and tracked beta support for v1.33; containerd support landed later through the containerd 2.x line. On anything before Kubernetes v1.36, verify both the Kubernetes feature gate and the runtime version before treating this as production plumbing.


Decision framework by Kubernetes version

K8s versionBuild images (CI)Mount image contents as volume
< 1.24Avoid Docker socket builds; use BuildKit, Buildah, or legacy Kaniko with explicit risk acceptanceInit container + emptyDir workaround
1.24 – 1.30BuildKit or Buildah preferred; legacy Kaniko only if already standardizedInit container + emptyDir workaround
1.31 – 1.32BuildKit or Buildah preferred; legacy Kaniko only with maintenance planImage Volumes alpha — ImageVolume feature gate required, runtime support required
1.33 – 1.34BuildKit or Buildah preferred; legacy Kaniko only with maintenance planImage Volumes beta with subPath/subPathExpr, but disabled by default; enable ImageVolume and verify runtime support
1.35BuildKit or Buildah preferred; legacy Kaniko only with maintenance planImage Volumes beta, enabled by default; still verify runtime support before production rollout
1.36+BuildKit or Buildah preferred; legacy Kaniko only for existing pipelines or maintained forksImage Volumes stable (GA), enabled by default

When to use what

Use Kaniko if: – You already have working Kaniko pipelines and the operational cost of migration is higher than the current risk – You have a maintained fork, vendor support, or an internal patching process – You want the simplest daemonless build setup and accept that upstream Google Kaniko is archived

Use BuildKit (rootless) if: – You need advanced cache mounts or multi-platform builds – Your team already uses docker buildx locally – You’re willing to run and operate BuildKit builders in the cluster

Use Buildah if: – You are on OpenShift or a Podman/Red Hat-oriented platform – You want an OCI-native build tool that does not require a daemon – Your cluster policy supports the user namespace requirements of rootless builds

Use Image Volumes if: – You want to inject versioned, signed artifacts into pods without building anything – You’re replacing init container + emptyDir patterns – You’re adopting OCI images as a general artifact format (configs, schemas, binaries)


Conclusion

The container image story inside Kubernetes has matured significantly. Docker-in-Docker is dead — correctly so. Kaniko solved an important build problem in 2018, but the original Google project is archived in 2026, so it should no longer be the default recommendation for new platforms. BuildKit and Buildah are the healthier starting points for active build pipelines, with Kaniko reserved for existing estates or explicitly supported forks.

Image Volumes are genuinely new ground. They’re not competing with Kaniko — they’re addressing a different layer of the same ecosystem: the distribution and consumption of OCI artifacts beyond just “images you run.” With GA in v1.36 and the supply chain tooling around OCI reaching maturity, the pattern of “package it as an image, distribute it like an image, mount it where you need it” is becoming the right answer for a class of problems that previously lived in ConfigMaps, PVCs, or init container hacks.

The right combination depends on what you’re doing and what version you’re running. But for the first time, Kubernetes has native answers for both sides of the equation.

Choose today

If you operate Kubernetes v1.36 or newer, use Image Volumes for read-only artifact injection and choose BuildKit or Buildah for builds. If you are on v1.35, Image Volumes are beta and enabled by default, but you should still verify runtime support and keep the init-container pattern as the rollback path. If you are on v1.33 or v1.34, beta does not mean default-on: enable the ImageVolume feature gate deliberately and validate the runtime. If you are on v1.31 or v1.32, treat Image Volumes as alpha and non-default. If you are below v1.31, Image Volumes are not part of the platform: use emptyDir plus an init container for artifact mounting, and modernize the build path separately.

For CI builds, start with BuildKit when you want Dockerfile compatibility, cache performance, multi-platform output, and a path aligned with docker buildx. Start with Buildah when your cluster is OpenShift or Podman-oriented. Keep Kaniko only where it already works and where someone owns the maintenance risk.

Sources

  • https://github.com/GoogleContainerTools/kaniko
  • https://github.com/kubernetes/enhancements/issues/4639
  • https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
  • https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-volume-source/
  • https://kubernetes.io/blog/2025/04/29/kubernetes-v1-33-image-volume-beta/
  • https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/
  • https://docs.docker.com/build/builders/drivers/kubernetes/
  • https://github.com/moby/buildkit/blob/master/docs/rootless.md
  • https://github.com/containers/buildah
  • https://github.com/containers/buildah/blob/main/docs/tutorials/05-openshift-rootless-build.md

Helm Chart Testing in Production: Layers, Tools, and a Minimum CI Pipeline

Helm Chart Testing in Production: Layers, Tools, and a Minimum CI Pipeline

When a Helm chart fails in production, the impact is immediate and visible. A misconfigured ServiceAccount, a typo in a ConfigMap key, or an untested conditional in templates can trigger incidents that cascade through your entire deployment pipeline. The irony is that most teams invest heavily in testing application code while treating Helm charts as “just configuration.”

Chart testing is fundamental for production-quality Helm deployments. For comprehensive coverage of testing along with all other Helm best practices, visit our complete Helm guide.

Helm charts are infrastructure code. They define how your applications run, scale, and integrate with the cluster. Treating them with less rigor than your application logic is a risk most production environments cannot afford.

The Real Cost of Untested Charts

In late 2024, a medium-sized SaaS company experienced a 4-hour outage because a chart update introduced a breaking change in RBAC permissions. The chart had been tested locally with helm install --dry-run, but the dry-run validation doesn’t interact with the API server’s RBAC layer. The deployment succeeded syntactically but failed operationally.

The incident revealed three gaps in their workflow:

  1. No schema validation against the target Kubernetes version
  2. No integration tests in a live cluster
  3. No policy enforcement for security baselines

These gaps are common. According to a 2024 CNCF survey on GitOps practices, fewer than 40% of organizations systematically test Helm charts before production deployment.

The problem is not a lack of tools—it’s understanding which layer each tool addresses.

Testing Layers: What Each Level Validates

Helm chart testing is not a single operation. It requires validation at multiple layers, each catching different classes of errors.

Layer 1: Syntax and Structure Validation

What it catches: Malformed YAML, invalid chart structure, missing required fields

Tools:

  • helm lint: Built-in, minimal validation following Helm best practices
  • yamllint: Strict YAML formatting rules

Example failure caught:

# Invalid indentation breaks the chart
resources:
  limits:
      cpu: "500m"
    memory: "512Mi"  # Incorrect indentation

Limitation: Does not validate whether the rendered manifests are valid Kubernetes objects.

Layer 2: Schema Validation

What it catches: Manifests that would be rejected by the Kubernetes API

Primary tool: kubeconform

Kubeconform is the actively maintained successor to the deprecated kubeval. It validates against OpenAPI schemas for specific Kubernetes versions and can include custom CRDs.

Project Profile:

  • Maintenance: Active, community-driven
  • Strengths: CRD support, multi-version validation, fast execution
  • Why it matters: helm lint validates chart structure, but not if rendered manifests match Kubernetes schemas

Example failure caught:

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: app
        image: nginx:latest
# Missing required field: spec.selector

Configuration example:

helm template my-chart . | kubeconform \
  -kubernetes-version 1.30.0 \
  -schema-location default \
  -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
  -summary

Example CI integration:

#!/bin/bash
set -e

KUBE_VERSION="1.30.0"

echo "Rendering chart..."
helm template my-release ./charts/my-chart > manifests.yaml

echo "Validating against Kubernetes $KUBE_VERSION..."
kubeconform \
  -kubernetes-version "$KUBE_VERSION" \
  -schema-location default \
  -summary \
  -output json \
  manifests.yaml | jq -e '.summary.invalid == 0'

Alternative: kubectl --dry-run=server (requires cluster access, validates against actual API server)

Layer 3: Unit Testing

What it catches: Logic errors in templates, incorrect conditionals, wrong value interpolation

Unit tests validate that given a set of input values, the chart produces the expected manifests. This is where template logic is verified before reaching a cluster.

Primary tool: helm-unittest

helm-unittest is the most widely adopted unit testing framework for Helm charts.

Project Profile:

  • GitHub: 3.3k+ stars, ~100 contributors
  • Maintenance: Active (releases every 2-3 months)
  • Primary maintainer: Quentin Machu (originally @QubitProducts, now independent)
  • Commercial backing: None
  • Bus Factor: Medium-High (no institutional backing, but consistent community engagement)

Strengths:

  • Fast execution (no cluster required)
  • Familiar test syntax (similar to Jest/Mocha)
  • Snapshot testing support
  • Good documentation

Limitations:

  • Doesn’t validate runtime behavior
  • Cannot test interactions with admission controllers
  • No validation against actual Kubernetes API

Example test scenario:

# tests/deployment_test.yaml
suite: test deployment
templates:
  - deployment.yaml
tests:
  - it: should set resource limits when provided
    set:
      resources.limits.cpu: "1000m"
      resources.limits.memory: "1Gi"
    asserts:
      - equal:
          path: spec.template.spec.containers[0].resources.limits.cpu
          value: "1000m"
      - equal:
          path: spec.template.spec.containers[0].resources.limits.memory
          value: "1Gi"

  - it: should not create HPA when autoscaling disabled
    set:
      autoscaling.enabled: false
    template: hpa.yaml
    asserts:
      - hasDocuments:
          count: 0

Alternative: Terratest (Helm module)

Terratest is a Go-based testing framework from Gruntwork that includes first-class Helm support. Unlike helm-unittest, Terratest deploys charts to real clusters and allows programmatic assertions in Go.

Example Terratest test:

func TestHelmChartDeployment(t *testing.T) {
    kubectlOptions := k8s.NewKubectlOptions("", "", "default")
    options := &helm.Options{
        KubectlOptions: kubectlOptions,
        SetValues: map[string]string{
            "replicaCount": "3",
        },
    }
    
    defer helm.Delete(t, options, "my-release", true)
    helm.Install(t, options, "../charts/my-chart", "my-release")
    
    k8s.WaitUntilNumPodsCreated(t, kubectlOptions, metav1.ListOptions{
        LabelSelector: "app=my-app",
    }, 3, 30, 10*time.Second)
}

When to use Terratest vs helm-unittest:

  • Use helm-unittest for fast, template-focused validation in CI
  • Use Terratest when you need full integration testing with Go flexibility

Layer 4: Integration Testing

What it catches: Runtime failures, resource conflicts, actual Kubernetes behavior

Integration tests deploy the chart to a real (or ephemeral) cluster and verify it works end-to-end.

Primary tool: chart-testing (ct)

chart-testing is the official Helm project for testing charts in live clusters.

Project Profile:

  • Ownership: Official Helm project (CNCF)
  • Maintainers: Helm team (contributors from Microsoft, IBM, Google)
  • Governance: CNCF-backed with public roadmap
  • LTS: Aligned with Helm release cycle
  • Bus Factor: Low (institutional backing from CNCF provides strong long-term guarantees)

Strengths:

  • De facto standard for public Helm charts
  • Built-in upgrade testing (validates migrations)
  • Detects which charts changed in a PR (efficient for monorepos)
  • Integration with GitHub Actions via official action

Limitations:

  • Requires a live Kubernetes cluster
  • Initial setup more complex than unit testing
  • Does not include security scanning

What ct validates:

  • Chart installs successfully
  • Upgrades work without breaking state
  • Linting passes
  • Version constraints are respected

Example ct configuration:

# ct.yaml
target-branch: main
chart-dirs:
  - charts
chart-repos:
  - bitnami=https://charts.bitnami.com/bitnami
helm-extra-args: --timeout 600s
check-version-increment: true

Typical GitHub Actions workflow:

name: Lint and Test Charts

on: pull_request

jobs:
  lint-test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Set up Helm
        uses: azure/setup-helm@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Set up chart-testing
        uses: helm/chart-testing-action@v2

      - name: Run chart-testing (lint)
        run: ct lint --config ct.yaml

      - name: Create kind cluster
        uses: helm/kind-action@v1

      - name: Run chart-testing (install)
        run: ct install --config ct.yaml

When ct is essential:

  • Public chart repositories (expected by community)
  • Charts with complex upgrade paths
  • Multi-chart repositories with CI optimization needs

Layer 5: Security and Policy Validation

What it catches: Security misconfigurations, policy violations, compliance issues

This layer prevents deploying charts that pass functional tests but violate organizational security baselines or contain vulnerabilities.

Policy Enforcement: Conftest (Open Policy Agent)

Conftest is the CLI interface to Open Policy Agent for policy-as-code validation.

Project Profile:

  • Parent: Open Policy Agent (CNCF Graduated Project)
  • Governance: Strong CNCF backing, multi-vendor support
  • Production adoption: Netflix, Pinterest, Goldman Sachs
  • Bus Factor: Low (graduated CNCF project with multi-vendor backing)

Strengths:

  • Policies written in Rego (reusable, composable)
  • Works with any YAML/JSON input (not Helm-specific)
  • Can enforce organizational standards programmatically
  • Integration with admission controllers (Gatekeeper)

Limitations:

  • Rego has a learning curve
  • Does not replace functional testing

Example Conftest policy:

# policy/security.rego
package main

import future.keywords.contains
import future.keywords.if
import future.keywords.in

deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not container.resources.limits.memory
  msg := sprintf("Container '%s' must define memory limits", [container.name])
}

deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not container.resources.limits.cpu
  msg := sprintf("Container '%s' must define CPU limits", [container.name])
}

Running the validation:

helm template my-chart . | conftest test -p policy/ -

Alternative: Kyverno

Kyverno offers policy enforcement using native Kubernetes manifests instead of Rego. Policies are written in YAML and can validate, mutate, or generate resources.

Example Kyverno policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-container-limits
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "All containers must have CPU and memory limits"
      pattern:
        spec:
          containers:
          - resources:
              limits:
                memory: "?*"
                cpu: "?*"

Conftest vs Kyverno:

  • Conftest: Policies run in CI, flexible for any YAML
  • Kyverno: Runtime enforcement in-cluster, Kubernetes-native

Both can coexist: Conftest in CI for early feedback, Kyverno in cluster for runtime enforcement.

Vulnerability Scanning: Trivy

Trivy by Aqua Security provides comprehensive security scanning for Helm charts.

Project Profile:

  • Maintainer: Aqua Security (commercial backing with open-source core)
  • Scope: Vulnerability scanning + misconfiguration detection
  • Helm integration: Official trivy helm command
  • Bus Factor: Low (commercial backing + strong open-source adoption)

What Trivy scans in Helm charts:

  1. Vulnerabilities in referenced container images
  2. Misconfigurations (similar to Conftest but pre-built rules)
  3. Secrets accidentally committed in templates

Example scan:

trivy helm ./charts/my-chart --severity HIGH,CRITICAL --exit-code 1

Sample output:

myapp/templates/deployment.yaml (helm)
====================================

Tests: 12 (SUCCESSES: 10, FAILURES: 2)
Failures: 2 (HIGH: 1, CRITICAL: 1)

HIGH: Container 'app' of Deployment 'myapp' should set 'securityContext.runAsNonRoot' to true
════════════════════════════════════════════════════════════════════════════════════════════════
Ensure containers run as non-root users

See https://kubernetes.io/docs/concepts/security/pod-security-standards/
────────────────────────────────────────────────────────────────────────────────────────────────
 myapp/templates/deployment.yaml:42

Commercial support:
Aqua Security offers Trivy Enterprise with advanced features (centralized scanning, compliance reporting). For most teams, the open-source version is sufficient.

Other Security Tools

Polaris (Fairwinds)

Polaris scores charts based on security and reliability best practices. Unlike enforcement tools, it provides a health score and actionable recommendations.

Use case: Dashboard for chart quality across a platform

Checkov (Bridgecrew/Palo Alto)

Similar to Trivy but with a broader IaC focus (Terraform, CloudFormation, Kubernetes, Helm). Pre-built policies for compliance frameworks (CIS, PCI-DSS).

When to use Checkov:

  • Multi-IaC environment (not just Helm)
  • Compliance-driven validation requirements

Enterprise Selection Criteria

Bus Factor and Long-Term Viability

For production infrastructure, tool sustainability matters as much as features. Community support channels like Helm CNCF Slack (#helm-users, #helm-dev) and CNCF TAG Security provide valuable insights into which projects have active maintainer communities.

Questions to ask:

  • Is the project backed by a foundation (CNCF, Linux Foundation)?
  • Are multiple companies contributing?
  • Is the project used in production by recognizable organizations?
  • Is there a public roadmap?

Risk Classification:

Tool Governance Bus Factor Notes
chart-testing CNCF Low Helm official project
Conftest/OPA CNCF Graduated Low Multi-vendor backing
Trivy Aqua Security Low Commercial backing + OSS
kubeconform Community Medium Active, but single maintainer
helm-unittest Community Medium-High No institutional backing
Polaris Fairwinds Medium Company-sponsored OSS

Kubernetes Version Compatibility

Tools must explicitly support the Kubernetes versions you run in production.

Red flags:

  • No documented compatibility matrix
  • Hard-coded dependencies on old K8s versions
  • No testing against multiple K8s versions in CI

Example compatibility check:

# Does the tool support your K8s version?
kubeconform --help | grep -A5 "kubernetes-version"

For tools like ct, always verify they test against a matrix of Kubernetes versions in their own CI.

Commercial Support Options

When commercial support matters:

  • Regulatory compliance requirements (SOC2, HIPAA, etc.)
  • Limited internal expertise
  • SLA-driven operations

Available options:

  • Trivy: Aqua Security offers Trivy Enterprise
  • OPA/Conftest: Styra provides OPA Enterprise
  • Terratest: Gruntwork offers consulting and premium modules

Most teams don’t need commercial support for chart testing specifically, but it’s valuable in regulated industries where audits require vendor SLAs.

Security Scanner Integration

For enterprise pipelines, chart testing tools should integrate cleanly with:

  • SIEM/SOAR platforms
  • CI/CD notification systems
  • Security dashboards (e.g., Grafana, Datadog)

Required features:

  • Structured output formats (JSON, SARIF)
  • Exit codes for CI failure
  • Support for custom policies
  • Webhook or API for event streaming

Example: Integrating Trivy with SIEM

# .github/workflows/security.yaml
- name: Run Trivy scan
  run: trivy helm ./charts --format json --output trivy-results.json

- name: Send to SIEM
  run: |
    curl -X POST https://siem.company.com/api/events \
      -H "Content-Type: application/json" \
      -d @trivy-results.json

Testing Pipeline Architecture

A production-grade Helm chart pipeline combines multiple layers:

Pipeline efficiency principles:

  1. Fail fast: syntax and schema errors should never reach integration tests
  2. Parallel execution where possible (unit tests + security scans)
  3. Cache ephemeral cluster images to reduce setup time
  4. Skip unchanged charts (ct built-in change detection)

Decision Matrix: When to Use What

Scenario 1: Small Team / Early-Stage Startup

Requirements: Minimal overhead, fast iteration, reasonable safety

Recommended Stack:

Linting:      helm lint + yamllint
Validation:   kubeconform
Security:     trivy helm

Optional: helm-unittest (if template logic becomes complex)

Rationale: Zero-dependency baseline that catches 80% of issues without operational complexity.

Scenario 2: Enterprise with Compliance Requirements

Requirements: Auditable, comprehensive validation, commercial support available

Recommended Stack:

Linting:      helm lint + yamllint
Validation:   kubeconform
Unit Tests:   helm-unittest
Security:     Trivy Enterprise + Conftest (custom policies)
Integration:  chart-testing (ct)
Runtime:      Kyverno (admission control)

Optional: Terratest for complex upgrade scenarios

Rationale: Multi-layer defense with both pre-deployment and runtime enforcement. Commercial support available for security components.

Scenario 3: Multi-Tenant Internal Platform

Requirements: Prevent bad charts from affecting other tenants, enforce standards at scale

Recommended Stack:

CI Pipeline:
  • helm lint → kubeconform → helm-unittest → ct
  • Conftest (enforce resource quotas, namespaces, network policies)
  • Trivy (block critical vulnerabilities)

Runtime:
  • Kyverno or Gatekeeper (enforce policies at admission)
  • ResourceQuotas per namespace
  • NetworkPolicies by default

Additional tooling:

  • Polaris dashboard for chart quality scoring
  • Custom admission webhooks for platform-specific rules

Rationale: Multi-tenant environments cannot tolerate “soft” validation. Runtime enforcement is mandatory.

Scenario 4: Open Source Public Charts

Requirements: Community trust, transparent testing, broad compatibility

Recommended Stack:

Must-have:
  • chart-testing (expected standard)
  • Public CI (GitHub Actions with full logs)
  • Test against multiple K8s versions

Nice-to-have:
  • helm-unittest with high coverage
  • Automated changelog generation
  • Example values for common scenarios

Rationale: Public charts are judged by testing transparency. Missing ct is a red flag for potential users.

The Minimum Viable Testing Stack

For any environment deploying Helm charts to production, this is the baseline:

Layer 1: Pre-Commit (Developer Laptop)

helm lint charts/my-chart
yamllint charts/my-chart

Layer 2: CI Pipeline (Automated on PR)

# Fast validation
helm template my-chart ./charts/my-chart | kubeconform \
  -kubernetes-version 1.30.0 \
  -summary

# Security baseline
trivy helm ./charts/my-chart --exit-code 1 --severity CRITICAL,HIGH

Layer 3: Pre-Production (Staging Environment)

# Integration test with real cluster
ct install --config ct.yaml --charts charts/my-chart

Time investment:

  • Initial setup: 4-8 hours
  • Per-PR overhead: 3-5 minutes
  • Maintenance: ~1 hour/month

ROI calculation:

Average production incident caused by untested chart:

  • Detection: 15 minutes
  • Triage: 30 minutes
  • Rollback: 20 minutes
  • Post-mortem: 1 hour
  • Total: ~2.5 hours of engineering time

If chart testing prevents even one incident per quarter, it pays for itself in the first month.

Common Anti-Patterns to Avoid

Anti-Pattern 1: Only using --dry-run

helm install --dry-run validates syntax but skips:

  • Admission controller logic
  • RBAC validation
  • Actual resource creation

Better: Combine dry-run with kubeconform and at least one integration test.

Anti-Pattern 2: Testing only in production-like clusters

“We test in staging, which is identical to production.”

Problem: Staging clusters rarely match production exactly (node counts, storage classes, network policies). Integration tests should run in isolated, ephemeral environments.

Anti-Pattern 3: Security scanning without enforcement

Running trivy helm without failing the build on critical findings is theater.

Better: Set --exit-code 1 and enforce in CI.

Anti-Pattern 4: Ignoring upgrade paths

Most chart failures happen during upgrades, not initial installs. Chart-testing addresses this with ct install --upgrade.

Conclusion: Testing is Infrastructure Maturity

The gap between teams that test Helm charts and those that don’t is not about tooling availability—it’s about treating infrastructure code with the same discipline as application code.

The cost of testing is measured in minutes per PR. The cost of not testing is measured in hours of production incidents, eroded trust in automation, and teams reverting to manual deployments because “Helm is too risky.”

The testing stack you choose matters less than the fact that you have one. Start with the minimal viable stack (lint + schema + security), run it consistently, and expand as your charts become more complex.

By implementing a structured testing pipeline, you catch 95% of chart issues before they reach production. The remaining 5% are edge cases that require production observability, not more testing layers.

Helm chart testing is not about achieving perfection—it’s about eliminating the preventable failures that undermine confidence in your deployment pipeline.

Frequently Asked Questions (FAQ)

What is Helm chart testing and why is it important in production?

Helm chart testing ensures that Kubernetes manifests generated from Helm templates are syntactically correct, schema-compliant, secure, and function correctly when deployed. In production, untested charts can cause outages, security incidents, or failed upgrades, even if application code itself is stable.

Is helm lint enough to validate a Helm chart?

No. helm lint only validates chart structure and basic best practices. It does not validate rendered manifests against Kubernetes API schemas, test template logic, or verify runtime behavior. Production-grade testing requires additional layers such as schema validation, unit tests, and integration tests.

What is the difference between Helm unit tests and integration tests?

Unit tests (e.g., using helm-unittest) validate template logic by asserting expected output for given input values without deploying anything. Integration tests (e.g., using chart-testing or Terratest) deploy charts to a real Kubernetes cluster and validate runtime behavior, upgrades, and interactions with the API server.

Which tools are recommended for validating Helm charts against Kubernetes schemas?

The most commonly recommended tool is kubeconform, which validates rendered manifests against Kubernetes OpenAPI schemas for specific Kubernetes versions and supports CRDs. An alternative is kubectl --dry-run=server, which validates against a live API server.

How can Helm chart testing prevent production outages?

Testing catches common failure modes before deployment, such as missing selectors in Deployments, invalid RBAC permissions, incorrect conditionals, or incompatible API versions. Many production outages originate from configuration and chart logic errors rather than application bugs.

What is the role of security scanning in Helm chart testing?

Security scanning detects misconfigurations, policy violations, and vulnerabilities that functional tests may miss. Tools like Trivy and Conftest (OPA) help enforce security baselines, prevent unsafe defaults, and block deployments that violate organizational or compliance requirements.

Is chart-testing (ct) required for private Helm charts?

While not strictly required, chart-testing is highly recommended for any chart deployed to production. It is considered the de facto standard for integration testing, especially for charts with upgrades, multiple dependencies, or shared cluster environments.

What is the minimum viable Helm testing pipeline for CI?

At a minimum, a production-ready pipeline should include:
helm lint for structural validation
kubeconform for schema validation
trivy helm for security scanning
Integration tests can be added as charts grow in complexity or criticality.