Alexandre Vazquez

The Problem: Docker Is Overkill for Image Operations

You need to copy an image from Docker Hub to your private registry. Or inspect a manifest before pulling. Or delete old tags programmatically. Or sync an entire repository during a migration.

Related reading: Kaniko, BuildKit and Image Volumes.

The instinct is to reach for Docker. But Docker requires a running daemon, root access (or group membership that amounts to the same thing), and pulls the entire image to disk just to read its metadata. For CI pipelines, GitOps workflows, and platform tooling, that’s a significant overhead for what should be lightweight registry operations.

This is the problem that daemonless container image tools solve. Skopeo pioneered the category; today it has real competition from crane, regctl, and ORAS — each with different strengths and ideal use cases.

This article gives you the practical comparison to pick the right tool for your workflow.

The Contenders

All five are Go binaries, statically compiled, and work directly against the OCI Distribution Spec. None of them require Docker or any container runtime.

Skopeo

Skopeo was the first major tool to address daemonless image operations, released by Red Hat in 2016 as part of the containers/image ecosystem (alongside Podman and Buildah).

What Skopeo does well

Image inspection without pulling:

skopeo inspect docker://registry.k8s.io/pause:3.9

Returns full image metadata — digest, layers, labels, architecture, OS — without downloading a single layer. Useful in admission controllers, policy checks, and pre-deployment validation.

Cross-registry copying:

skopeo copy 
  docker://docker.io/library/nginx:1.27 
  docker://harbor.internal/library/nginx:1.27

Copies image manifests and layers directly between registries, bypassing your local machine entirely. The image never touches your disk.

Multi-arch handling:

skopeo copy --all 
  docker://docker.io/library/nginx:1.27 
  docker://harbor.internal/library/nginx:1.27

The --all flag copies the full manifest list, preserving all architectures (linux/amd64, linux/arm64, etc.). This is critical when mirroring images for multi-arch clusters.

Registry synchronization:

skopeo sync 
  --src docker 
  --dest docker 
  --all 
  docker.io/library/nginx 
  harbor.internal/mirrors/

skopeo sync mirrors an entire repository, including all tags. You can also use a YAML file to define which images and tags to sync — useful for air-gapped environment bootstrapping.

Tag deletion:

skopeo delete docker://harbor.internal/myapp:old-tag

Useful in CI for cleanup pipelines. Note that registry-side deletion requires the registry to have the DELETE method enabled.

Skopeo’s weaknesses

• No image modification: Skopeo copies and inspects, but doesn’t build or modify images
• Tag listing is verbose: skopeo list-tags returns JSON you need to parse
• No retry logic by default: transient network errors in long sync operations require wrapping with retry scripts
• Auth configuration: relies on containers/auth.json format, which differs from Docker’s ~/.docker/config.json (though it supports both)

When to use Skopeo

• Air-gapped environment image mirroring
• CI pipelines that need to copy or inspect images without Docker
• Platform teams on Red Hat / OpenShift stacks
• Any workflow already using Podman or Buildah

Crane

Crane is Google’s answer to Skopeo, developed as part of the ko project and later extracted into its own tool. It’s simpler, more scriptable, and has a cleaner CLI design.

What Crane does well

Tag listing:

crane ls registry.k8s.io/pause

No JSON parsing needed. One tag per line. Pipe directly into grep, sort, head.

Digest resolution:

crane digest docker.io/library/nginx:1.27

Returns the image digest. Combine with yq or sed to pin image references in Helm values or Kubernetes manifests.

Image copying:

crane cp docker.io/library/nginx:1.27 harbor.internal/library/nginx:1.27

Same capability as Skopeo’s copy, arguably with a cleaner syntax.

Manifest inspection:

crane manifest docker.io/library/nginx:1.27 | jq .

Returns raw manifest JSON. Useful when you need the exact manifest for digest verification or policy enforcement.

Tagging and retagging:

crane tag harbor.internal/myapp:abc123 harbor.internal/myapp:stable

Adds a new tag to an existing image without re-uploading layers. The tag operation is purely a manifest pointer update.

Flattening images:

crane flatten docker.io/library/ubuntu:24.04 -t harbor.internal/ubuntu:flat

Squashes all layers into one. Reduces layer count for images where layer history doesn’t matter.

Crane’s weaknesses

• No sync command: unlike Skopeo, crane has no built-in repository sync. You script it yourself with crane ls + crane cp in a loop
• Less mature multi-arch support: crane cp supports multi-arch but the UX is less explicit than Skopeo’s --all
• No delete command: doesn’t implement registry deletion

When to use Crane

• CI/CD scripting where you want clean, pipeable output
• Digest pinning workflows
• Lightweight image tagging operations
• When you’re already in the ko / Google Cloud ecosystem

regctl

Regctl is the least known of the three but arguably the most feature-complete. It’s the CLI for the regclient Go library and covers use cases that Skopeo and crane leave out.

What regctl does uniquely well

Image modification without rebuild:

regctl image mod myimage:tag 
  --label "org.opencontainers.image.version=1.2.3" 
  --replace

You can add/change labels, annotations, and config fields directly on an existing image in the registry — without pulling, rebuilding, or pushing a new image. This is impossible with Skopeo or crane.

Layer operations:

# Remove a specific layer from an image
regctl image mod myimage:tag 
  --layer-rm sha256:abc123... 
  --replace

Useful for removing accidentally included secrets or large unnecessary layers from published images.

OCI artifact support:

regctl artifact put 
  --media-type application/vnd.example.config.v1+json 
  --config config.json 
  file.tar.gz 
  harbor.internal/myartifacts:v1

Regctl has solid OCI artifact support alongside standard image operations.

Formatting and output:

regctl tag list harbor.internal/myapp --format '{{range .}}{{println .}}{{end}}'

Go template formatting throughout. Useful for integrating into shell scripts without jq.

Referrers (OCI 1.1):

regctl manifest get-list harbor.internal/myapp:v1 --referrers

Lists referrers (signatures, SBOMs, attestations) attached to an image via the OCI 1.1 referrers API.

Regctl’s weaknesses

• Smaller community: fewer examples, less StackOverflow coverage
• Steeper learning curve: more commands, more flags
• Less packaging: not in most distro repos by default

When to use regctl

• Image post-processing (labels, annotations, layer removal) without rebuild
• Advanced manifest and referrer workflows
• When you need OCI artifact operations alongside image operations

ORAS

ORAS (OCI Registry As Storage) is a CNCF project focused specifically on OCI artifact management — pushing and pulling arbitrary files to container registries, not necessarily container images.

# Push a Helm chart as an OCI artifact
oras push harbor.internal/charts/myapp:1.0.0 
  --artifact-type application/vnd.helm.chart.v1+tar 
  mychart.tgz

# Push SBOM
oras push harbor.internal/myapp:v1 
  --artifact-type application/spdx+json 
  sbom.spdx.json

# Pull
oras pull harbor.internal/charts/myapp:1.0.0

ORAS is not a direct Skopeo replacement — it’s for when your registry is a general-purpose artifact store, not just a container registry. Helm OCI, SBOMs, attestations, and policy bundles all benefit from ORAS.

ORAS vs Skopeo in one line: Skopeo (and crane) move container images between registries; ORAS pushes and pulls arbitrary artifacts (charts, SBOMs, ML models) to a registry. If your object is an OCI image, use Skopeo or crane. If it is a file you want to store in a registry, use ORAS. They overlap far less than the shared “registry client” label suggests, and most platform teams end up with both installed for different jobs.

cosign

Cosign from Sigstore is not a general-purpose image tool — it’s specifically for supply chain security. But it’s increasingly part of any container image workflow.

# Sign an image
cosign sign --key cosign.key harbor.internal/myapp:v1@sha256:abc123...

# Verify
cosign verify --key cosign.pub harbor.internal/myapp:v1

# Attach SBOM
cosign attach sbom --sbom sbom.spdx harbor.internal/myapp:v1

# Keyless signing (Sigstore)
cosign sign harbor.internal/myapp:v1

Cosign integrates with OIDC providers for keyless signing (no key management required), which is the direction the ecosystem is moving. If you’re building a supply chain security practice, cosign is mandatory, not optional.

Side-by-side comparison

Practical workflows

Copy an image from Docker Hub to a private registry (crane cp vs skopeo copy)

This is the single most common reason people reach for these tools, so here are the two one-liners side by side. Both copy the image directly registry-to-registry — the image never touches your local disk and no Docker daemon is involved.

# skopeo copy — explicit docker:// transport on both sides
skopeo copy
  docker://docker.io/library/nginx:1.27
  docker://harbor.internal/library/nginx:1.27

# crane cp — shorter, no transport prefix
crane cp docker.io/library/nginx:1.27 harbor.internal/library/nginx:1.27

They are functionally equivalent for a single image. Reach for skopeo copy when you want --all to force the full multi-arch manifest list, or when you are on a Red Hat / Podman stack. Reach for crane cp when you want the shortest possible command and cleaner output for scripting — it copies the manifest list automatically when the source is multi-arch. To authenticate against the private registry first, both read ~/.docker/config.json, so a prior docker login harbor.internal (or crane auth login / skopeo login) is enough. To copy many images or whole repositories in one go, jump to the air-gapped mirroring recipe below, which uses skopeo sync.

Mirror images for air-gapped clusters (Skopeo)

# sync-list.yaml
docker.io:
  images:
    library/nginx:
      - "1.25"
      - "1.26"
      - "1.27"
    library/redis:
      - "7.2"
      - "7.4"

skopeo sync 
  --src yaml 
  --dest docker 
  --all 
  sync-list.yaml 
  harbor.internal/mirrors/

Pin image digests in CI (Crane)

#!/bin/bash
# Update image digests in values.yaml
for image in nginx:1.27 redis:7.4; do
  digest=$(crane digest docker.io/library/${image})
  echo "docker.io/library/${image}@${digest}"
done

Combine with yq to update Helm values files automatically, ensuring reproducible deployments.

Retag without re-pushing (Crane or regctl)

# After a successful deploy to staging, promote to production
crane tag harbor.internal/myapp:${GIT_SHA} harbor.internal/myapp:production

No layer transfer. The operation is a metadata update in the registry.

Add OCI annotations post-build (regctl)

regctl image mod harbor.internal/myapp:v1.2.3 
  --annotation "org.opencontainers.image.source=https://github.com/org/repo" 
  --annotation "org.opencontainers.image.revision=${GIT_SHA}" 
  --replace

Attaches build metadata to an image already in the registry, without a rebuild.

Supply chain security pipeline

# 1. Build and push
docker buildx build --push -t harbor.internal/myapp:${GIT_SHA} .

# 2. Generate SBOM
syft harbor.internal/myapp:${GIT_SHA} -o spdx-json > sbom.spdx.json

# 3. Attach SBOM
cosign attach sbom --sbom sbom.spdx.json harbor.internal/myapp:${GIT_SHA}

# 4. Sign (keyless with OIDC in CI)
cosign sign harbor.internal/myapp:${GIT_SHA}

# 5. Verify in admission controller or deployment pipeline
cosign verify 
  --certificate-identity-regexp="https://github.com/org/repo" 
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" 
  harbor.internal/myapp:${GIT_SHA}

Installation

All tools install as single static binaries:

# Skopeo (via package manager)
brew install skopeo                    # macOS
dnf install skopeo                     # RHEL/Fedora
apt install skopeo                     # Debian/Ubuntu

# Crane
brew install crane
# or binary release
curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz crane

# regctl
curl -sL https://github.com/regclient/regclient/releases/latest/download/regctl.linux.amd64 -o /usr/local/bin/regctl
chmod +x /usr/local/bin/regctl

# ORAS
brew install oras

# cosign
brew install cosign

Which tool should you use?

Use Skopeo if: you’re on a Red Hat / OpenShift stack, you need repository sync, or you’re building air-gapped environment pipelines. It’s the most battle-tested and widely packaged.

Use Crane if: you’re scripting image operations in CI and want clean, composable CLI output. crane ls + crane cp + crane digest cover 80% of automation use cases with minimal friction.

Use regctl if: you need to modify images post-build, work with OCI referrers, or want the most complete feature set for a registry client. It has a higher learning curve but can replace both Skopeo and crane for advanced workflows.

Use ORAS if: you’re using a registry to store non-image artifacts — Helm charts, SBOMs, policy bundles, ML models.

Use cosign regardless of which of the above you pick, as soon as supply chain security matters to your organization. It’s not a replacement for the others — it’s a complement.

In practice, most platform teams end up using 2-3 of these tools together. Crane for day-to-day scripting, Skopeo for sync jobs, cosign for signing, ORAS for artifact storage.

FAQ

Related reading: FreeLens vs OpenLens vs Lens: which one to standardize on.

The State of Kubernetes Desktop Tooling in 2026

• Lens — the original. Electron-based, polished, now commercial (Mirantis). The free Personal tier is non-commercial only. Pro is ~$22-35/user/month.
• OpenLens — the community fork of Lens before Mirantis closed exec/logs/shell in v6.3 (January 2023). Maintenance has slowed significantly. No active release cadence.
• FreeLens — a more active community fork, filling the gap left by OpenLens’ decline. Restores the missing features. No commercial backing.
• k9s — terminal TUI, fast, keyboard-driven, single-cluster. Different audience.
• Headlamp — CNCF Sandbox project, plugin-extensible, web-based.
• Radar — Go binary, Apache 2.0, team-oriented, topology and event timeline focused.

What Radar Actually Is

• Radar OSS — a single ~30MB Go binary, Apache 2.0, free forever. Can run locally (desktop app) or deployed in-cluster via Helm. No sidecars, no feature gates.
• Radar Cloud — same binary, adds a hosted control plane with fleet aggregation, 30-day event retention, SSO/SCIM, scoped RBAC, and shared URLs for team incident response. Priced per cluster ($99/cluster/month for Team), not per user.

Key Features

Topology View

Persistent Event Timeline

GitOps Integration (ArgoCD + Flux)

Helm Management

Image Filesystem

MCP Server (AI Integration)

Cluster Audit

Multi-Cluster Support (Cloud)

Architecture: Why a Go Binary Matters

• 200-500MB install size
• 1-2 second startup time on a fast machine, more on slower ones
• Memory footprint in the hundreds of megabytes
• Local kubeconfig required on each engineer’s machine

Feature Comparison

When Radar Makes Sense

When OpenLens or FreeLens Still Makes Sense

Getting Started

# Homebrew (macOS/Linux)
brew install skyhook-io/tap/radar

# Helm (in-cluster)
helm repo add skyhook https://charts.skyhook.io
helm install radar skyhook/radar 
  --namespace radar 
  --create-namespace 
  --set service.type=ClusterIP

Verdict

Related reading: FreeLens extensions: the complete catalogue and how to install them.

Karpenter vs Cluster Autoscaler: the short answer

Why Node Autoscaling Is Hard

1. Over-provisioning — you run enough nodes to handle peak load at all times. Your average utilization sits at 20–30%, and you’re paying for the other 70–80% to sit idle.
2. Under-provisioning — you run lean, and when traffic spikes, pods go Pending. Your SLOs breach. You get paged at 3am to manually scale.

Cluster Autoscaler: How It Actually Works

The Node Group Model

Scale-Up: Detecting Unschedulable Pods

• least-waste — minimizes CPU/memory waste after scheduling (best default for cost)
• most-pods — maximizes pods scheduled per scale-up operation
• priority — lets you define ordering via ConfigMap
• grpc — delegates to an external gRPC service

# Cluster Autoscaler deployment — AWS, production-tuned
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: cluster-autoscaler
      containers:
        - image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.36.1
          name: cluster-autoscaler
          resources:
            requests:
              cpu: 100m
              memory: 600Mi
            limits:
              cpu: 200m
              memory: 1Gi
          command:
            - ./cluster-autoscaler
            - --cloud-provider=aws
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster
            - --expander=least-waste
            - --balance-similar-node-groups=true
            - --scale-down-delay-after-add=10m
            - --scale-down-unneeded-time=10m
            - --scale-down-utilization-threshold=0.5
            - --max-graceful-termination-sec=600
            - --scan-interval=10s

Scale-Down: The Conservative Approach

Karpenter: How It Actually Works

The Core Insight: Bypass the Node Group

NodePool and EC2NodeClass

# EC2NodeClass — cloud-specific parameters
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  role: "KarpenterNodeRole-my-cluster"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 50Gi
        volumeType: gp3
        encrypted: true
  metadataOptions:
    httpTokens: required
    httpPutResponseHopLimit: 1
---
# NodePool — intent and constraints
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-size
          operator: NotIn
          values: ["nano", "micro", "small", "medium", "large"]
      expireAfter: 720h
  limits:
    cpu: "1000"
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets:
      - nodes: "5%"
        schedule: "0 8 * * mon-fri"
        duration: 10h
      - nodes: "25%"

Just-in-Time Provisioning and Bin Packing

Disruption and Consolidation

Architecture Comparison

Scaling Speed: The Numbers

Cost Optimization: Where Karpenter Pulls Ahead

What’s New in Karpenter v1.14 (July 2026)

Where that leaves the release line today: v1.14.1 (21 August 2026) is the current release and is tagged as an LTS line by the AWS provider, with support committed through July 2027; the previous LTS is v1.9.x. There is no v1.15 yet. If you are choosing a version to standardise on for the next year, v1.14 is the one — it has the DRA allocator, CapacityBuffer and the Balanced policy, and it will keep receiving patches after the interim minors are dropped.

Related reading: what CNCF maturity levels actually mean.

Related reading: Kubernetes HPA memory examples.

What’s New in Cluster Autoscaler (2026)

Multi-Cloud Support in 2026

Karpenter vs HPA, VPA and KEDA: Which Autoscaler Does What

A recurring confusion in the comparison searches: Karpenter is not an alternative to HPA or KEDA. They operate on different layers and are designed to run together. Pod autoscalers decide how many replicas you need; node autoscalers decide what hardware those replicas land on. Karpenter and Cluster Autoscaler only ever see the outcome of the pod autoscalers — pending pods.

So “KEDA vs Karpenter” and “HPA vs Karpenter” are not real choices. KEDA and HPA answer how many pods; Karpenter and Cluster Autoscaler answer how many nodes, and which ones. You will almost always run one from each row. The real comparison is HPA vs KEDA on the pod layer (KEDA when the signal is a queue, a topic or a schedule rather than CPU) and Karpenter vs Cluster Autoscaler on the node layer, which is what the rest of this article is about.

The chain in practice

A queue backs up, KEDA sees the lag and raises the replica count, the new pods cannot be scheduled because the cluster is full, and Karpenter provisions a node that fits them. Every layer does its own job. The failure mode people hit is tuning only one of them: an HPA that scales aggressively on a cluster with a slow node autoscaler produces pods that sit pending for minutes, and the latency you were trying to fix stays exactly where it was.

Two pairings deserve care. HPA and VPA on the same metric fight each other — VPA raises requests, which lowers measured CPU utilisation, which makes HPA scale down; run VPA in Off or Initial mode if an HPA already targets CPU on that workload. And KEDA scaling to zero only works if the node autoscaler can drain the last node: with Karpenter, that is consolidation doing its job; with Cluster Autoscaler, you need the node group’s minimum set to 0. See HPA best practices for the pod-level half of this.

Replacing Cluster Autoscaler with Karpenter

If the goal is a straight replacement rather than coexistence, the order matters: install Karpenter with a NodePool that is restricted to the workloads you are moving, scale the Cluster Autoscaler node groups down as Karpenter takes over, and only then remove the Cluster Autoscaler deployment. Removing it first leaves the old node groups with nobody managing scale-down, and you pay for idle nodes until someone notices.

When Cluster Autoscaler Is Still the Right Choice

1. Non-AWS environments — GCP, Alibaba, DigitalOcean, on-prem with Cluster API
2. Existing node group architecture — significant investment in ASG design, compliance tooling
3. Regulatory constraints — some frameworks require ASG-backed provisioning audit trails
4. Cluster API / bare metal — CA is the only mature option
5. Team familiarity and working-well CA deployment — migration cost may not justify benefit

When Karpenter Is the Right Choice

1. AWS-native, cost optimization priority — right-sizing + consolidation = meaningful cost reduction
2. Diverse and variable workloads — batch, spot, GPU, stateless APIs — Karpenter handles all with a few NodePools
3. Spot-heavy clusters — native interruption handling, diversification, no NTH
4. Declarative infrastructure-as-code culture — NodePools version cleanly in Git
5. Low-latency scaling requirements — event-driven workloads, KEDA-triggered jobs, sharp traffic spikes

Running Both: Migration Path and Gotchas

Separating Responsibility

# NodePool with taint — CA-managed pods won't tolerate this
spec:
  template:
    metadata:
      labels:
        provisioner: karpenter
    spec:
      taints:
        - key: karpenter.sh/provisioned
          value: "true"
          effect: NoSchedule

Gradual Migration

1. Phase 1 — Karpenter manages spot/batch workloads. CA manages on-demand production nodes.
2. Phase 2 — Migrate spot workloads fully. Remove AWS NTH.
3. Phase 3 — Migrate on-demand. Reduce CA node group capacity gradually.
4. Phase 4 — Decommission CA once all groups are empty.

Key Gotchas

• Karpenter consolidation + permissive PDBs — maxUnavailable: 100% will cause disruptive consolidation. Audit PDBs before enabling WhenEmptyOrUnderutilized.
• NodePool limits are hard stops — pods go Pending indefinitely at limit. Monitor utilization.
• AMI drift — @latest alias picks up new AMIs on new nodes. Consider pinning for strict change control.
• Simultaneous scale-down conflicts — use strict label/taint segregation during migration.

Karpenter vs Cluster Autoscaler Cost: When the Savings Are Real

• Instance selection across the whole catalog. A pending pod that needs 3 vCPU and 12 GiB lands on the cheapest instance that fits it — possibly a type you would never have created a node group for. With CA you pay for the granularity of the node groups you happened to define. This is the source that CA structurally cannot match, and in heterogeneous clusters it is the largest one.
• Active consolidation. Karpenter continuously replaces an underused large node with a smaller one that still fits every pod. CA’s scale-down is binary — a node is either below the utilisation threshold and removed, or left alone — so a cluster of 60%-utilised nodes never gets cheaper under CA and keeps shrinking under Karpenter.
• Spot without ceremony. Diversified spot requests, interruption handling and replacement provisioning are built in. CA can run spot node groups, but the diversification and the graceful replacement are on you (or on the Node Termination Handler).

• Homogeneous, reserved capacity. If your fleet is 90% one instance family covered by Savings Plans or Reserved Instances, the cheapest node is the one you already committed to. Karpenter’s catalog-wide bin packing has nothing to optimise, and its consolidation can actually hurt by moving load off committed capacity onto on-demand instances unless you constrain the NodePool tightly.
• Steady-state workloads. Consolidation saves money on clusters whose shape changes. A cluster that runs the same 40 pods all day at stable requests has no fragmentation to recover.
• Well-tuned least-waste with a small set of right-sized node groups. Most of the overprovisioning CA gets blamed for is a node-group design problem — three sizes per family, least-waste as the expander and --scale-down-utilization-threshold raised from the default 0.5 close most of the gap on simple clusters.

Decision Framework

Running on AWS?
├── No → Azure? → Karpenter (stable) or CA
│        GCP?   → CA or GKE NAP (preferred)
│        Other  → Cluster Autoscaler
│
└── Yes → Hard regulatory constraints on non-ASG provisioning?
          ├── Yes → Cluster Autoscaler
          └── No → Cost optimization priority or diverse workloads?
                   ├── Yes → Karpenter
                   └── No → Either (flip for team preference)

Frequently Asked Questions

Karpenter vs Cluster Autoscaler: Quick Decision Recap

Searching for Karpenter vs Cluster Autoscaler? The one-paragraph version: choose Karpenter on AWS when provisioning speed (60–90 seconds vs 4–8 minutes), instance-type flexibility and cost consolidation matter. Choose Cluster Autoscaler when you need multi-cloud portability, you already operate stable node groups, or your platform depends on the node-group model (compliance-pinned AMIs, strict capacity planning).

Related reading: GPU scheduling with Dynamic Resource Allocation.

Related reading: Kamal vs Kubernetes.

Related reading: EKS Auto Mode.

Both can coexist during a migration: run Cluster Autoscaler for existing node groups while Karpenter handles new dynamic capacity, then consolidate once you trust the disruption behavior.

What Requests and Limits Actually Are

CPU vs Memory: Why They Behave Fundamentally Differently

CPU Is Compressible

Memory Is Not Compressible

QoS Classes: Eviction Priority Under Pressure

Guaranteed

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Burstable

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "1000m"
    memory: "1Gi"

BestEffort

# No resources block at all

Common Misconfiguration Patterns and Their Consequences

Pattern 1: No Requests or Limits Set

Pattern 2: Requests Equal Limits (Guaranteed QoS)

Pattern 3: Limits Much Higher Than Requests (Burstable with High Ratio)

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "4000m"
    memory: "4Gi"

Pattern 4: CPU Limits Set to “Be Safe”

The CPU Throttling Problem: CFS Bandwidth and Hidden Latency

How CFS Bandwidth Throttling Works

• cpu.cfs_period_us: the accounting period, default 100ms
• cpu.cfs_quota_us: how many microseconds of CPU time the cgroup can use per period

Why This Causes Latency Spikes Even at Low Utilization

The Cloudflare and Netflix Evidence

The Recommendation: Consider Not Setting CPU Limits

Memory: Always Set Limits

Vertical Pod Autoscaler (VPA)

How VPA Works

• Recommender: Watches historical metrics and computes recommended requests based on observed utilization. Does not modify pods.
• Updater: Evicts pods whose current requests differ significantly from recommendations (when VPA mode is Auto or Recreate).
• Admission Controller: Mutates pod specs at admission time to apply recommendations from the Recommender.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-server-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Off"   # Recommend only — do not evict pods
  resourcePolicy:
    containerPolicies:
    - containerName: api-server
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 4000m
        memory: 4Gi
      controlledResources: ["cpu", "memory"]
      controlledValues: RequestsAndLimits

VPA Modes

When to Use VPA

LimitRange and ResourceQuota: Namespace-Level Guardrails

LimitRange

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
  - type: Container
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    max:
      cpu: "4000m"
      memory: "8Gi"
    min:
      cpu: "50m"
      memory: "64Mi"
  - type: Pod
    max:
      cpu: "8000m"
      memory: "16Gi"
  - type: PersistentVolumeClaim
    max:
      storage: "50Gi"
    min:
      storage: "1Gi"

ResourceQuota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    requests.cpu: "20"
    requests.memory: "40Gi"
    limits.cpu: "40"
    limits.memory: "80Gi"
    pods: "100"
    persistentvolumeclaims: "20"
    requests.storage: "500Gi"
    count/deployments.apps: "50"
    count/services: "50"
    count/secrets: "100"
    count/configmaps: "100"

Practical Sizing Methodology

Step 1: Instrument Before You Set Values

# p95 CPU usage over the last 7 days
histogram_quantile(0.95,
  rate(container_cpu_usage_seconds_total{
    container="api-server",
    namespace="production"
  }[5m])
)

# p99 memory working set over the last 7 days
quantile_over_time(0.99,
  container_memory_working_set_bytes{
    container="api-server",
    namespace="production"
  }[7d]
)

# CPU throttling ratio (alert if >5%)
rate(container_cpu_cfs_throttled_seconds_total{container="api-server"}[5m])
/
rate(container_cpu_cfs_periods_total{container="api-server"}[5m])

Step 2: Set CPU Requests from p95 Observations

Step 3: Set Memory Requests and Limits

# Production example: Java microservice
resources:
  requests:
    cpu: "500m"       # p95 observed: ~420m
    memory: "768Mi"   # p95 observed: ~680Mi
  limits:
    # No CPU limit — latency-sensitive service
    memory: "1.5Gi"   # 2x request, covers GC pressure

Step 4: Use VPA Recommendations to Validate

Step 5: Adjust for Workload Lifecycle Events

Decision Framework: What to Set Based on Workload Type

Monitoring and Alerting

# OOMKill rate
- alert: ContainerOOMKilled
  expr: increase(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[5m]) > 0
  for: 0m
  labels:
    severity: warning

# CPU throttling >10%
- alert: CPUThrottlingHigh
  expr: |
    rate(container_cpu_cfs_throttled_seconds_total[5m])
    /
    rate(container_cpu_cfs_periods_total[5m])
    > 0.10
  for: 5m
  labels:
    severity: warning

# Memory near limit >85%
- alert: MemoryNearLimit
  expr: |
    container_memory_working_set_bytes
    /
    (container_spec_memory_limit_bytes > 0)
    > 0.85
  for: 5m
  labels:
    severity: warning

FAQ

Seven years is a long time in observability. Since Prometheus 2.0 landed in 2017, the ecosystem has been transformed by cloud-native adoption, the rise of distributed tracing, and the emergence of OpenTelemetry as the de facto standard for instrumentation. Prometheus 3.0, released in November 2024, is the project’s answer to that transformation — and its most significant change is the native ability to ingest OpenTelemetry metrics directly, without an intermediary collector standing in the way.

Related reading: Prometheus cardinality and scaling.

This article goes deep on what Prometheus 3.0 actually changes for platform engineers and cloud architects who are running — or planning to run — OTel-instrumented workloads alongside Prometheus-based monitoring stacks. We will cover the native OTLP ingestion endpoint, UTF-8 metric name support, Remote Write 2.0, migration considerations, and the architectural patterns that still make sense even when native OTLP is available.

What Changed in Prometheus 3.0: The OTel-Relevant Picture

Prometheus 3.0 ships a substantial set of changes. Not all of them are equally relevant to OpenTelemetry integration, so let’s focus on what actually moves the needle for OTel users before diving into each area in detail.

Native OTLP Ingestion

The flagship feature: Prometheus 3.0 ships with a built-in OTLP receiver that exposes an HTTP endpoint accepting metrics in the OpenTelemetry Protocol format. Applications instrumented with any OTel SDK can now push metrics directly to Prometheus without routing through an OpenTelemetry Collector. This is not a sidecar, not a plugin, not an external adapter — it is a first-class endpoint in the Prometheus binary itself.

UTF-8 Metric Names

Prometheus historically restricted metric names to [a-zA-Z_:][a-zA-Z0-9_:]*. OpenTelemetry uses dots and slashes in metric names by convention — http.server.request.duration is a canonical OTel metric name. Prometheus 3.0 lifts this restriction and supports arbitrary UTF-8 characters in metric names and label names, which is the single most important compatibility change for OTel interoperability.

Remote Write 2.0

Remote Write 2.0

Remote Write 2.0 replaces the original protocol with a more efficient encoding based on protobuf, adds native histogram support in the wire format, and reduces bandwidth consumption significantly for large-scale deployments. If you are federating metrics to Thanos, Mimir, or Cortex, this matters for operational cost.

New UI

The Prometheus web UI has been completely rewritten. The new UI uses React, supports metric metadata exploration, and provides a significantly improved query-building experience. This is a quality-of-life improvement rather than an architectural change, but it reduces the dependency on external tools like Grafana for ad-hoc investigation.

Breaking Changes Summary

Prometheus 3.0 removes several features that were deprecated in 2.x. The most operationally significant are: removal of the --web.enable-admin-api deprecated flag path, removal of certain legacy storage format options, changes to default scrape timeouts, and stricter validation of configuration that was previously silently accepted. We cover a migration checklist later in this article.

The OTLP Receiver: How It Works and What It Accepts

The OTLP receiver in Prometheus 3.0 is implemented as an optional feature that must be explicitly enabled. Once enabled, it exposes an HTTP endpoint at /api/v1/otlp/v1/metrics that accepts protobuf-encoded OTLP ExportMetricsServiceRequest payloads — the same wire format used by the OpenTelemetry Collector’s OTLP exporter.

What It Accepts (and What It Does Not)

This is critical to understand before you architect around native OTLP ingestion: Prometheus 3.0 OTLP support is metrics-only. It does not accept traces or logs. OTLP is a unified protocol covering all three signals, but Prometheus is a metrics store — the receiver handles only the metrics portion of the OTLP specification.

Supported metric types in the OTLP receiver:

• Gauge — maps directly to a Prometheus Gauge
• Sum (monotonic) — maps to a Prometheus Counter
• Sum (non-monotonic) — maps to a Prometheus Gauge
• Histogram (explicit bucket) — maps to a Prometheus Histogram
• ExponentialHistogram — maps to Prometheus Native Histograms (a 3.0 feature)
• Summary — maps to a Prometheus Summary

Resource attributes from the OTLP payload — things like service.name, k8s.pod.name, cloud.region — are converted to Prometheus labels. This conversion is configurable, and by default Prometheus applies a promotion strategy that converts the most common resource attributes to labels while discarding ones that would create extremely high cardinality.

Enabling the OTLP Receiver

Enabling native OTLP ingestion requires two things: a feature flag and a configuration block in prometheus.yml.

Start the Prometheus binary with the feature flag:

prometheus 
  --config.file=/etc/prometheus/prometheus.yml 
  --enable-feature=otlp-write-receiver

Then add the OTLP receiver configuration to your prometheus.yml:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

otlp:
  # Promote these OTLP resource attributes to Prometheus labels
  promote_resource_attributes:
    - service.name
    - service.namespace
    - service.instance.id
    - k8s.namespace.name
    - k8s.pod.name
    - k8s.node.name
    - cloud.region
    - deployment.environment

With this configuration, Prometheus will listen on port 9090 (default) and accept OTLP metrics at http://<prometheus-host>:9090/api/v1/otlp/v1/metrics.

Resource Attribute Promotion Strategy

The promote_resource_attributes list deserves careful thought. OTLP carries rich resource-level context — every metric payload includes a ResourceMetrics object with attributes describing the source: service name, version, environment, Kubernetes pod, node, cluster, cloud provider details, and more. Prometheus labels are flat key-value pairs on each time series. Promoting too many resource attributes explodes cardinality; promoting too few loses important context.

A pragmatic starting list for Kubernetes deployments:

otlp:
  promote_resource_attributes:
    - service.name          # Critical: identifies the service
    - service.namespace     # Logical grouping
    - deployment.environment  # prod/staging/dev
    - k8s.namespace.name    # Kubernetes namespace
    - k8s.pod.name          # Pod-level cardinality — consider omitting in high-scale
    - k8s.node.name         # Useful for infrastructure correlation

Avoid blindly promoting k8s.pod.name at scale — in a cluster with thousands of short-lived pods, this creates significant cardinality pressure. Prefer service.name and service.namespace for most alerting use cases, reserving pod-level labels for debugging dashboards.

UTF-8 Metric Names: Why This Is the Real Game-Changer

To appreciate why UTF-8 metric name support matters so much, you need to understand the friction it eliminates. OpenTelemetry semantic conventions define metric names using dots as namespace separators. The canonical HTTP server duration metric is http.server.request.duration. The canonical database query duration is db.client.operation.duration. These names are standardized across languages and frameworks — your Go service and your Java service and your Python service all emit the same metric name when instrumented with OTel.

Prometheus 2.x could not store these names. The dots are illegal characters in Prometheus metric naming. Every OTel-to-Prometheus bridge — the OpenTelemetry Collector’s Prometheus exporter, prom-client compatibility layers, the older prometheusremotewrite exporter — had to translate these names, typically by replacing dots with underscores: http_server_request_duration.

This translation is lossy and creates multiple problems:

• Name collisions: http.server.request_duration and http.server.request.duration both become http_server_request_duration
• Dashboard breakage: Grafana dashboards built against OTel semantic conventions don’t work against translated Prometheus metrics without modification
• Cross-signal correlation: Trace attributes use dot notation; when metric names differ, automated correlation tools lose the thread
• Vendor lock-in pressure: Teams end up with separate naming conventions for “Prometheus metrics” vs “OTel metrics” and maintain both

Prometheus 3.0 with UTF-8 support stores http.server.request.duration natively. No translation. No collision. The metric name you instrument with is the metric name you query.

Enabling UTF-8 Metric Names

UTF-8 metric names require the utf8-names feature flag:

prometheus 
  --config.file=/etc/prometheus/prometheus.yml 
  --enable-feature=utf8-names 
  --enable-feature=otlp-write-receiver

Once enabled, PromQL queries must use quoted metric names when the name contains characters outside the legacy character set:

# Legacy metric name — unquoted works fine
http_server_requests_total

# OTel metric name with dots — requires quoting in PromQL
{"__name__"="http.server.request.duration"}

# Or using the new PromQL syntax in Prometheus 3.0
http.server.request.duration{service_name="api-gateway"}

The PromQL parser in Prometheus 3.0 has been updated to handle quoted metric names as a first-class construct. Grafana’s PromQL engine has also been updated to handle this syntax — verify your Grafana version (10.3+ has full support) before deploying.

OTel SDK to Prometheus 3.0 Directly: No Collector Required

For teams that only need to get application metrics into Prometheus, native OTLP ingestion enables a dramatically simpler architecture. Here’s what it looks like with different OTel SDKs.

Go (OpenTelemetry SDK)

package main

import (
    "context"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
    "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func initMetrics(ctx context.Context) (*metric.MeterProvider, error) {
    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName("my-api"),
            semconv.ServiceNamespace("platform"),
            semconv.DeploymentEnvironment("production"),
        ),
    )
    if err != nil {
        return nil, err
    }

    // Point directly at Prometheus 3.0 OTLP endpoint
    exporter, err := otlpmetrichttp.New(ctx,
        otlpmetrichttp.WithEndpoint("prometheus:9090"),
        otlpmetrichttp.WithURLPath("/api/v1/otlp/v1/metrics"),
        otlpmetrichttp.WithInsecure(), // Use WithTLSClientConfig for production
    )
    if err != nil {
        return nil, err
    }

    provider := metric.NewMeterProvider(
        metric.WithResource(res),
        metric.WithReader(
            metric.NewPeriodicReader(exporter,
                metric.WithInterval(30*time.Second),
            ),
        ),
    )

    otel.SetMeterProvider(provider)
    return provider, nil
}

Python (OpenTelemetry SDK)

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_NAMESPACE

resource = Resource.create({
    SERVICE_NAME: "my-api",
    SERVICE_NAMESPACE: "platform",
    "deployment.environment": "production",
})

exporter = OTLPMetricExporter(
    endpoint="http://prometheus:9090/api/v1/otlp/v1/metrics",
)

reader = PeriodicExportingMetricReader(
    exporter,
    export_interval_millis=30_000,
)

provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(provider)

# Use the meter
meter = metrics.get_meter("my-api")
request_counter = meter.create_counter(
    name="http.server.request.count",
    description="Total HTTP server requests",
    unit="1",
)
request_duration = meter.create_histogram(
    name="http.server.request.duration",
    description="HTTP server request duration",
    unit="s",
)

Java (OpenTelemetry SDK with Spring Boot)

# application.properties (Spring Boot with OTel auto-instrumentation)
otel.service.name=my-api
otel.resource.attributes=service.namespace=platform,deployment.environment=production

# Configure OTLP exporter to push directly to Prometheus
otel.metrics.exporter=otlp
otel.exporter.otlp.metrics.endpoint=http://prometheus:9090/api/v1/otlp/v1/metrics
otel.exporter.otlp.metrics.protocol=http/protobuf

# Export interval
otel.metric.export.interval=30000

With Spring Boot and the OTel Java agent, no code changes are required beyond configuration — the agent instruments your HTTP server, database clients, and messaging systems automatically and pushes metrics using the names defined in OTel semantic conventions.

OTel Collector to Prometheus 3.0: When You Need the Intermediary

Native OTLP ingestion is compelling, but the OpenTelemetry Collector remains relevant for a significant set of use cases. Understanding when each pattern is appropriate is the core architectural decision you will face when adopting Prometheus 3.0 in an OTel environment.

Pattern 1: OTel Collector as Fan-Out Gateway

When you need to send metrics to multiple backends simultaneously — Prometheus for alerting, a long-term store like Thanos for historical analysis, and a commercial observability platform for full-stack correlation — the OTel Collector handles fan-out efficiently. Applications push once to the Collector; the Collector distributes to all backends.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1000
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  # Push to Prometheus 3.0 via OTLP
  otlphttp/prometheus:
    endpoint: http://prometheus:9090/api/v1/otlp
    tls:
      insecure: true

  # Fan-out to Thanos via remote_write
  prometheusremotewrite/thanos:
    endpoint: http://thanos-receive:10908/api/v1/receive
    resource_to_telemetry_conversion:
      enabled: true

  # Fan-out to commercial backend
  otlp/datadog:
    endpoint: https://otel-intake.datadoghq.com
    headers:
      DD-API-KEY: "${DD_API_KEY}"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/prometheus, prometheusremotewrite/thanos, otlp/datadog]

Pattern 2: Collector for Metric Transformation

The OTel Collector’s transform processor and metricstransform processor allow you to reshape metrics before they reach Prometheus: rename labels, add static attributes, filter out high-cardinality series, aggregate metrics to reduce storage cost, or apply unit conversions. These operations are not available in Prometheus’s native OTLP receiver.

processors:
  transform/metrics:
    metric_statements:
      - context: metric
        statements:
          # Drop internal debug metrics
          - delete_matching_keys(attributes, "internal.*")
          # Normalize environment label values
          - set(attributes["deployment.environment"], "prod")
            where attributes["deployment.environment"] == "production"

  filter/drop_debug:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - ".*.debug..*"
          - "runtime.go.internal..*"

  metricstransform:
    transforms:
      # Rename a metric to match your existing Prometheus naming convention
      - include: http.server.request.duration
        action: update
        new_name: http_server_request_duration_seconds

Pattern 3: Collector for Traces and Logs (Always Required)

If your architecture includes traces and logs alongside metrics — and in 2025 it almost certainly does — you need an OTel Collector regardless of what you do with metrics. Prometheus does not accept traces or logs. Jaeger, Tempo, and Loki all have their own ingestion protocols. The Collector is the universal routing layer for the three pillars of observability.

In this architecture, it is usually simpler to route all three signals through the Collector and let it push metrics to Prometheus via OTLP or remote_write, rather than splitting metrics to go directly and everything else through the Collector.

When to Use Native OTLP vs. OTel Collector: Decision Framework

The pragmatic recommendation for most platform engineering teams: if you are already running the OTel Collector (and you should be if traces are in scope), continue routing metrics through it. Use the Collector’s otlphttp exporter to push to Prometheus 3.0. Reserve the direct SDK-to-Prometheus pattern for simple services where the Collector would be the only reason to add complexity.

Remote Write 2.0: What Changes for Existing Setups

Remote Write 2.0 is a significant protocol upgrade with real operational implications for teams using Prometheus as a metrics source for long-term storage systems like Thanos, Mimir, VictoriaMetrics, or Cortex.

Key Protocol Changes

• Protobuf encoding with snappy compression — replacing the previous text-based format. Typically 50-70% reduction in wire size for large metric batches
• Native histogram support in the wire format — exponential histograms can now be forwarded without converting to classic histograms, preserving full resolution
• Metadata forwarding — metric type and unit information is now transmitted alongside samples, enabling better downstream processing
• Created timestamps — the timestamp at which a counter was created is forwarded, enabling more accurate rate calculations across restarts

Configuring Remote Write 2.0

# prometheus.yml
remote_write:
  - url: "http://thanos-receive:10908/api/v1/receive"
    # Remote Write 2.0 is negotiated automatically with compatible receivers
    # Force RW2.0 explicitly if needed:
    send_native_histograms: true
    metadata_config:
      send: true
      send_interval: 1m
    queue_config:
      capacity: 10000
      max_shards: 200
      max_samples_per_send: 2000
      batch_send_deadline: 5s

Remote Write 2.0 uses protocol content negotiation — Prometheus 3.0 will attempt RW2.0 first and fall back to RW1.0 if the receiver does not support it. This means upgrades are generally backward-compatible. Verify that your receiving system (Thanos Receive 0.35+, Mimir 2.12+, VictoriaMetrics 1.98+) supports RW2.0 before expecting the benefits.

Migration from Prometheus 2.x: Breaking Changes Checklist

Upgrading from Prometheus 2.x to 3.0 requires attention to several breaking changes. This checklist covers the operationally significant ones for teams running production Prometheus deployments.

Configuration Changes

• Removed: query.lookback-delta default change — the default changed from 5 minutes to match the scrape interval. Queries that relied on the 5m default may return different results. Audit alerting rules that use instant queries on counters.
• Removed: deprecated remote_write options — remote_write[].queue_config.capacity semantics changed. Review and update queue configurations.
• Removed: storage.tsdb.allow-overlapping-blocks flag — overlapping blocks handling is now automatic. Remove this flag from your startup scripts.
• Scrape protocols default change — Prometheus 3.0 defaults to OpenMetrics format for scraping when targets support it. This enables native histograms but may surface parsing differences. Test with --enable-feature=no-default-scrape-port removed if you relied on the old behavior.
• Agent mode changes — if using Prometheus Agent mode, review the updated configuration options for WAL management.

PromQL Changes

• Stricter parsing — some previously accepted but technically invalid PromQL expressions now fail. Run your alerting rules through promtool check rules against a Prometheus 3.0 binary before cutover.
• Native histogram functions — new functions like histogram_fraction() and histogram_quantile() have updated behavior with native histograms. Existing dashboard queries using histogram_quantile() on classic histograms continue to work unchanged.

Storage Compatibility

Prometheus 3.0 can read existing 2.x TSDB data. The upgrade path does not require a data migration. However, Prometheus 2.x cannot read data blocks written by 3.0 (downgrade is not supported without data loss after any writes have occurred). Take a snapshot before upgrading if you need rollback capability:

# Take a TSDB snapshot before upgrading
curl -X POST http://prometheus:9090/api/v1/admin/tsdb/snapshot

# Verify the snapshot exists
ls /prometheus/snapshots/

Pre-Upgrade Validation Steps

# 1. Validate configuration against Prometheus 3.0
docker run --rm -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml 
  prom/prometheus:v3.0.0 
  promtool check config /etc/prometheus/prometheus.yml

# 2. Validate alerting rules
docker run --rm -v $(pwd)/rules:/etc/prometheus/rules 
  prom/prometheus:v3.0.0 
  promtool check rules /etc/prometheus/rules/*.yml

# 3. Run in parallel (shadow mode) before full cutover
# Deploy Prometheus 3.0 alongside 2.x, scraping the same targets
# Compare query results between versions using promtool query range

Practical Kubernetes Deployment Example

Here is a production-ready Kubernetes deployment of Prometheus 3.0 with OTLP ingestion enabled, suitable as a starting point for platform engineering teams.

Prometheus 3.0 ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
      external_labels:
        cluster: production
        region: eu-west-1

    otlp:
      promote_resource_attributes:
        - service.name
        - service.namespace
        - deployment.environment
        - k8s.namespace.name
        - k8s.pod.name

    rule_files:
      - /etc/prometheus/rules/*.yml

    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: "true"
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)

    remote_write:
      - url: http://thanos-receive.monitoring.svc.cluster.local:10908/api/v1/receive
        send_native_histograms: true
        metadata_config:
          send: true

Prometheus 3.0 Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
          image: prom/prometheus:v3.0.0
          args:
            - --config.file=/etc/prometheus/prometheus.yml
            - --storage.tsdb.path=/prometheus/data
            - --storage.tsdb.retention.time=15d
            - --web.enable-lifecycle
            - --web.enable-admin-api
            - --enable-feature=otlp-write-receiver
            - --enable-feature=utf8-names
            - --enable-feature=native-histograms
          ports:
            - name: http
              containerPort: 9090
              protocol: TCP
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
            - name: data
              mountPath: /prometheus/data
          resources:
            requests:
              cpu: 500m
              memory: 2Gi
            limits:
              cpu: 2000m
              memory: 8Gi
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: http
            initialDelaySeconds: 30
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /-/ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: config
          configMap:
            name: prometheus-config
        - name: data
          persistentVolumeClaim:
            claimName: prometheus-data
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
    - name: http
      port: 9090
      targetPort: http
  type: ClusterIP

Configuring Applications to Push OTLP

With this deployment, any application in the cluster can push OTLP metrics by setting the following environment variables (works with any OTel SDK supporting OTLP HTTP):

env:
  - name: OTEL_SERVICE_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.labels['app']
  - name: OTEL_SERVICE_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: OTEL_METRICS_EXPORTER
    value: "otlp"
  - name: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
    value: "http://prometheus.monitoring.svc.cluster.local:9090/api/v1/otlp/v1/metrics"
  - name: OTEL_EXPORTER_OTLP_METRICS_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_METRIC_EXPORT_INTERVAL
    value: "30000"
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "deployment.environment=production,k8s.namespace.name=$(NAMESPACE)"

This approach works particularly well in environments using the OTel Operator for Kubernetes, where the Instrumentation CRD can inject these environment variables automatically into pods based on namespace or pod label selectors — zero-touch instrumentation with native Prometheus storage.

Frequently Asked Questions

Can I use Prometheus 3.0 OTLP ingestion for traces and logs?

No. The Prometheus 3.0 OTLP receiver handles only metrics. Prometheus is a metrics store — it has no data model for traces or logs. For traces, you need a backend like Jaeger or Grafana Tempo. For logs, you need Loki, Elasticsearch, or a similar system. The OTel Collector is the appropriate routing layer when you need to send all three signals to their respective backends from a single application-side push endpoint.

Does the kube-prometheus-stack Helm chart support Prometheus 3.0?

Yes, with caveats. The kube-prometheus-stack chart updated its Prometheus image to 3.0 starting with chart version 66.0.0. However, some bundled recording rules and alerting rules may need adjustment for the PromQL changes and default behavioral differences. The Prometheus Operator itself (version 0.78+) has been updated to support the new configuration options including the otlp configuration block. If you are managing Prometheus via the Operator, you will configure OTLP settings through the Prometheus CRD’s spec.additionalArgs and a custom PrometheusConfiguration resource.

What happens to existing Prometheus 2.x metric names when I enable UTF-8 support?

Existing metrics with underscore-based names continue to work exactly as before. Enabling UTF-8 support is purely additive — it allows the storage and querying of metric names containing dots and other UTF-8 characters, but it does not rename or modify existing metrics. Your existing dashboards, alerting rules, and recording rules continue to function without modification. Only metrics ingested via OTLP (or exposed by exporters using OTel naming conventions) will use dot-separated names.

How does native OTLP ingestion affect Prometheus’s pull model?

It coexists with it. Prometheus 3.0 continues to scrape targets via the pull model on the same 9090 port. The OTLP endpoint is an additional ingestion path, not a replacement for scraping. You can have a Prometheus instance simultaneously scraping Kubernetes pods via service discovery and receiving OTLP push metrics from applications — both are stored in the same TSDB and queryable via the same PromQL interface. This hybrid approach is common during migrations, where legacy components are scraped and new OTel-instrumented services push via OTLP.

Is the Prometheus 3.0 OTLP receiver suitable for high-volume production workloads?

For moderate volumes, yes. The OTLP receiver is synchronous — the HTTP request completes only after the samples are written to the WAL. Under very high ingestion rates (hundreds of thousands of samples per second), this can create back-pressure that affects application latency. The OTel Collector handles this better through internal buffering, retry queues, and batch processing. For high-volume scenarios, the recommended pattern is: applications push to OTel Collector (which acknowledges immediately and buffers), Collector pushes to Prometheus via OTLP or remote_write in optimized batches. For the majority of Kubernetes workloads — dozens to hundreds of services with typical metric cardinality — the native OTLP receiver performs well without an intermediary.

Frequently Asked Questions

Helm is the de facto package manager for Kubernetes, and values.yaml is its primary interface for configuration. Yet for years, that interface has been completely unvalidated by default — a free-form YAML file where any key can be anything, where typos silently pass through, and where misconfigured deployments only reveal themselves when pods fail to start in production. The values.schema.json file changes that equation entirely. This article explains why schema validation matters, how to implement it properly, and how to integrate it into a modern CI/CD pipeline.

The Problem: Silent Failures in Production

Consider a platform team managing dozens of Helm releases across multiple clusters. A developer submits a values override file with replicaCount: "3" instead of replicaCount: 3 — a string where an integer is expected. Or they set image.pullPolicy: Allways with a typo. Or they omit a required secret reference that the application needs to boot. In all three cases, Helm without schema validation will happily render the templates, produce Kubernetes manifests, and apply them to the cluster. The failure surfaces later — sometimes much later — as a CrashLoopBackOff, an ImagePullBackOff, or a subtle runtime error that takes hours to debug.

This is not a hypothetical scenario. It is the daily reality for teams operating at scale without values validation. The root cause is architectural: Helm templates use Go’s text/template engine, which is weakly typed and permissive by design. A template that does {{ .Values.replicaCount }} will render whether the value is an integer, a string, or even a boolean. The resulting Kubernetes manifest may be invalid, but that error only surfaces when the Kubernetes API server rejects it — or worse, accepts it but interprets it differently than intended.

The consequences compound at scale. When a chart is used by multiple teams, the lack of a formal contract for acceptable values means every consumer has to read through template files and comments to understand what inputs are valid. There is no machine-readable specification. There is no IDE support. There is no guardrail. The only documentation is whatever the chart author happened to write in comments inside values.yaml — and comments do not stop a CI pipeline from shipping a broken deployment.

What Is values.schema.json

Since Helm 3.0.0, released in November 2019, Helm supports an optional values.schema.json file at the root of a chart directory — the same level as Chart.yaml and values.yaml. This file is a JSON Schema draft-07 document that formally describes the structure, types, constraints, and required fields for the chart’s values.

When this file is present, Helm automatically validates the merged values (defaults from values.yaml merged with any user-supplied overrides) against the schema at multiple points: during helm install, helm upgrade, helm template, and helm lint. If validation fails, Helm refuses to proceed and prints a human-readable error message identifying exactly which value failed and why. This transforms a class of runtime failures into build-time failures — the correct direction for any production system.

The choice of JSON Schema draft-07 specifically is worth noting. Draft-07 is widely supported by tooling, including the Red Hat YAML extension for VS Code, JetBrains IDEs, and most JSON Schema validators. It introduced the if/then/else conditional keywords that are particularly useful for Helm charts. More recent drafts (2019-09, 2020-12) offer additional features but have less universal tooling support, making draft-07 the pragmatic choice for chart authors today.

Chart Directory Structure

my-app/
├── Chart.yaml
├── values.yaml
├── values.schema.json      ← lives here
├── charts/
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── ingress.yaml
    └── _helpers.tpl

The schema file is included when a chart is packaged with helm package and distributed through chart repositories. Consumers of the chart get schema validation automatically without any additional configuration — the guardrails ship with the chart itself.

How Helm Uses the Schema

Helm’s validation behavior is straightforward but has some nuances worth understanding. When Helm processes a release, it first merges all value sources in order of increasing precedence: chart defaults (values.yaml), parent chart values, -f value files, and finally --set flags. The merged result is then validated against the schema as a single operation.

This means the schema validates the effective values, not each source in isolation. A required field that has a default in values.yaml will pass validation even when not specified by the user, because the merged result includes the default. This is the correct behavior — it validates what will actually be used during rendering.

The validation happens before template rendering. If schema validation fails, Helm exits with a non-zero status code and prints all validation errors. The error output is structured and actionable:

$ helm install my-release ./my-app --set replicaCount=abc

Error: values don't meet the specifications of the schema(s) in the following chart(s):
my-app:
- replicaCount: Invalid type. Expected: integer, given: string

For helm lint, which is typically used in CI pipelines without installing to a cluster, schema validation also runs. This makes helm lint a powerful pre-deployment gate when schema files are present.

IDE Benefits: Autocompletion and Inline Validation

Beyond Helm’s own validation, values.schema.json unlocks IDE support that significantly improves the developer experience when working with values files. The Red Hat YAML extension for VS Code can reference a JSON Schema file to provide autocompletion, type checking, and inline error highlighting for YAML files.

To enable this, add a yaml.schemas configuration to your VS Code workspace settings or the user settings file:

// .vscode/settings.json
{
  "yaml.schemas": {
    "./my-app/values.schema.json": "./my-app/values.yaml"
  }
}

With this configuration, editing values.yaml in VS Code will show autocompletion for defined keys, inline errors for type mismatches, and hover documentation pulled from the description fields in your schema. For platform teams maintaining internal Helm charts, this transforms the chart into a self-documenting, IDE-aware configuration interface — without any additional tooling investment.

JetBrains IDEs (IntelliJ IDEA, GoLand, etc.) support JSON Schema associations through the Languages & Frameworks > Schemas and DTDs > JSON Schema Mappings settings panel, providing equivalent functionality for teams using those tools.

Building the Schema: A Practical Guide

Let’s build a complete, realistic example. Start with a typical values.yaml for a web application chart:

# values.yaml
replicaCount: 2

image:
  repository: myorg/my-app
  tag: "1.0.0"
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80

ingress:
  enabled: false
  hostname: ""
  tls: false

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80

config:
  logLevel: info
  databaseUrl: ""

nodeSelector: {}
tolerations: []
affinity: {}

Now the full values.schema.json that validates this structure:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "my-app Helm Chart Values",
  "description": "Configuration values for the my-app Helm chart",
  "type": "object",
  "additionalProperties": false,
  "required": ["image", "service"],
  "$defs": {
    "resourceQuantity": {
      "type": "string",
      "pattern": "^[0-9]+(\\.[0-9]+)?(m|Ki|Mi|Gi|Ti|Pi|Ei|k|M|G|T|P|E)?$",
      "description": "A Kubernetes resource quantity (e.g. 100m, 128Mi, 1Gi)"
    }
  },
  "properties": {
    "replicaCount": {
      "type": "integer",
      "minimum": 0,
      "maximum": 50,
      "default": 2,
      "description": "Number of pod replicas. Set to 0 to scale down."
    },
    "image": {
      "type": "object",
      "additionalProperties": false,
      "required": ["repository", "tag"],
      "description": "Container image configuration",
      "properties": {
        "repository": {
          "type": "string",
          "minLength": 1,
          "description": "Container image repository"
        },
        "tag": {
          "type": "string",
          "pattern": "^[a-zA-Z0-9._-]+$",
          "minLength": 1,
          "description": "Image tag. Avoid using 'latest' in production."
        },
        "pullPolicy": {
          "type": "string",
          "enum": ["Always", "IfNotPresent", "Never"],
          "default": "IfNotPresent",
          "description": "Kubernetes imagePullPolicy"
        }
      }
    },
    "service": {
      "type": "object",
      "additionalProperties": false,
      "required": ["type", "port"],
      "properties": {
        "type": {
          "type": "string",
          "enum": ["ClusterIP", "NodePort", "LoadBalancer", "ExternalName"],
          "description": "Kubernetes Service type"
        },
        "port": {
          "type": "integer",
          "minimum": 1,
          "maximum": 65535,
          "description": "Service port"
        }
      }
    },
    "ingress": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "enabled": {
          "type": "boolean",
          "default": false
        },
        "hostname": {
          "type": "string",
          "description": "Ingress hostname. Required when ingress.enabled is true."
        },
        "tls": {
          "type": "boolean",
          "default": false,
          "description": "Enable TLS for the ingress"
        }
      },
      "if": {
        "properties": {
          "enabled": { "const": true }
        },
        "required": ["enabled"]
      },
      "then": {
        "required": ["hostname"],
        "properties": {
          "hostname": {
            "minLength": 1,
            "pattern": "^[a-zA-Z0-9]([a-zA-Z0-9\\-\\.]+)?[a-zA-Z0-9]$"
          }
        }
      }
    },
    "resources": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "requests": {
          "type": "object",
          "additionalProperties": false,
          "properties": {
            "cpu": { "$ref": "#/$defs/resourceQuantity" },
            "memory": { "$ref": "#/$defs/resourceQuantity" }
          }
        },
        "limits": {
          "type": "object",
          "additionalProperties": false,
          "properties": {
            "cpu": { "$ref": "#/$defs/resourceQuantity" },
            "memory": { "$ref": "#/$defs/resourceQuantity" }
          }
        }
      }
    },
    "autoscaling": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "enabled": {
          "type": "boolean",
          "default": false
        },
        "minReplicas": {
          "type": "integer",
          "minimum": 1
        },
        "maxReplicas": {
          "type": "integer",
          "minimum": 1,
          "maximum": 100
        },
        "targetCPUUtilizationPercentage": {
          "type": "integer",
          "minimum": 1,
          "maximum": 100
        }
      },
      "if": {
        "properties": {
          "enabled": { "const": true }
        },
        "required": ["enabled"]
      },
      "then": {
        "required": ["minReplicas", "maxReplicas"]
      }
    },
    "config": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "logLevel": {
          "type": "string",
          "enum": ["debug", "info", "warn", "error"],
          "default": "info",
          "description": "Application log level"
        },
        "databaseUrl": {
          "type": "string",
          "description": "Database connection URL"
        }
      }
    },
    "nodeSelector": {
      "type": "object",
      "description": "Node selector labels for pod scheduling"
    },
    "tolerations": {
      "type": "array",
      "description": "Pod tolerations"
    },
    "affinity": {
      "type": "object",
      "description": "Pod affinity rules"
    }
  }
}

Key Schema Patterns Explained

additionalProperties: false

This is arguably the most important pattern in a Helm schema. Without it, unknown keys pass validation silently — which defeats much of the purpose. With "additionalProperties": false, any key not listed in properties causes a validation error. This catches typos like repicaCount instead of replicaCount, which would otherwise silently use the default value and leave the developer wondering why their override had no effect.

Apply it at every nested object level, not just the root. A typo inside image: or resources: is just as dangerous as one at the top level.

$defs for Reusable Definitions

The $defs keyword (called definitions in earlier draft versions, though draft-07 supports both) provides a namespace for reusable schema fragments. In the example above, resourceQuantity is defined once and referenced via $ref in both requests and limits. This avoids duplication and ensures consistent validation logic across related fields.

For larger charts, $defs becomes essential. Common patterns include reusable schemas for image configurations, resource requirements, probe configurations, and environment variable maps.

Conditional Validation with if/then/else

The if/then/else construct in JSON Schema draft-07 is particularly powerful for Helm charts, where many values are conditional on a feature toggle. The ingress example above demonstrates this: when ingress.enabled is true, the hostname field becomes required and must match a valid hostname pattern. When ingress is disabled, the hostname can be empty or omitted entirely.

This pattern can be extended for more complex scenarios. For example, enforcing that when autoscaling.enabled is true, the standalone replicaCount should not be set (since the HPA controls replica count):

{
  "if": {
    "properties": {
      "autoscaling": {
        "properties": {
          "enabled": { "const": true }
        },
        "required": ["enabled"]
      }
    }
  },
  "then": {
    "properties": {
      "replicaCount": {
        "description": "replicaCount is ignored when autoscaling is enabled"
      }
    }
  }
}

Pattern Validation for Image Tags

The image tag field is a common source of production issues. Teams accidentally deploy with latest, which is non-deterministic and makes rollbacks unreliable. A pattern constraint can enforce semantic versioning or at least ban the latest tag in production charts:

"tag": {
  "type": "string",
  "not": {
    "enum": ["latest", ""]
  },
  "pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+",
  "description": "Semantic version tag required. 'latest' is not permitted."
}

This enforces that image tags start with a semantic version number, immediately rejecting latest, empty strings, or arbitrary branch names that would produce non-reproducible deployments.

Enum for Controlled Vocabularies

Fields with a fixed set of valid values — Kubernetes service types, image pull policies, log levels — should use enum. This is more precise than a pattern and produces clearer error messages. It also enables IDE autocompletion to show exactly the valid options as a pick-list, rather than requiring the developer to remember or look up acceptable values.

CI/CD Integration

GitHub Actions

The most direct integration point is helm lint, which runs schema validation as part of its checks. A minimal GitHub Actions workflow that validates a chart on every pull request looks like this:

# .github/workflows/helm-lint.yaml
name: Helm Lint

on:
  pull_request:
    paths:
      - 'charts/**'

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Helm
        uses: azure/setup-helm@v4
        with:
          version: '3.14.0'

      - name: Lint chart with default values
        run: helm lint charts/my-app

      - name: Lint chart with staging values
        run: helm lint charts/my-app -f charts/my-app/ci/staging-values.yaml

      - name: Lint chart with production values
        run: helm lint charts/my-app -f charts/my-app/ci/production-values.yaml

      - name: Validate template rendering
        run: |
          helm template my-app charts/my-app \
            -f charts/my-app/ci/production-values.yaml \
            --debug > /dev/null

The ci/ directory convention (values files specifically for CI testing) is a pattern from the chart-testing tool and works well for validating multiple realistic value combinations, not just the defaults.

For teams using the ct (chart-testing) CLI tool from the Helm project, schema validation is automatically included in the ct lint command, which also handles chart versioning checks and YAML linting:

      - name: Chart Testing lint
        uses: helm/chart-testing-action@v2.6.1

      - name: Run chart-testing lint
        run: ct lint --target-branch ${{ github.event.repository.default_branch }}

Pre-commit Hooks

For local development, pre-commit hooks catch issues before code is even pushed. The pre-commit framework makes this straightforward:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gruntwork-io/pre-commit
    rev: v0.1.23
    hooks:
      - id: helmlint

  - repo: local
    hooks:
      - id: helm-schema-validate
        name: Helm Schema Validation
        language: script
        entry: scripts/validate-helm-schemas.sh
        files: ^charts/.*values.*\.yaml$

#!/usr/bin/env bash
# scripts/validate-helm-schemas.sh
set -euo pipefail

for chart_dir in charts/*/; do
  if [[ -f "${chart_dir}/values.schema.json" ]]; then
    echo "Linting ${chart_dir}..."
    helm lint "${chart_dir}" --strict
  fi
done

ArgoCD and Flux Integration

Both ArgoCD and Flux (Helm Controller) invoke helm template internally when reconciling Helm releases. Since helm template runs schema validation when a schema file is present, any invalid values in an HelmRelease or ArgoCD Application manifest will cause the reconciliation to fail with a clear error message — visible in the controller logs and surface as a degraded resource status. No additional configuration is required; schema validation is automatic.

Generating Schemas from Existing Charts

For charts that already have a well-structured values.yaml, writing a schema from scratch is time-consuming but not starting from zero. Several tools can generate a draft schema that you then refine:

• helm-values-schema-json — a Helm plugin (helm plugin install https://github.com/losisin/helm-values-schema-json) that introspects values.yaml and generates a draft schema with inferred types. Run with helm schema-gen values.yaml.
• json-schema-generator online tools — paste your values as JSON (convert YAML to JSON first) and get a draft schema back.
• Manually from scratch — for new charts, writing the schema alongside the values file from the beginning is the most accurate approach and requires no extra tooling.

Generated schemas are always starting points. They infer types from existing values but cannot know about intended constraints, enums, patterns, required fields in conditional cases, or additionalProperties: false at nested levels. Manual review and refinement is always necessary.

Common Mistakes and How to Avoid Them

The Null Value Problem

A subtle but common issue: in YAML, an unset key with no value (key:) resolves to null, not an empty string or zero. If your schema defines a field as "type": "string", a null value will fail validation. To handle optional fields that users might leave blank, use a type union:

"databaseUrl": {
  "type": ["string", "null"],
  "description": "Database connection URL. Leave null to use the default."
}

Alternatively, ensure your values.yaml defaults use empty strings ("") rather than bare keys, and document that convention for chart consumers.

Schema Drift

As charts evolve, new values get added to values.yaml without corresponding updates to values.schema.json. Over time the schema becomes stale and provides partial coverage. The fix is procedural: treat schema updates as part of the definition of done for any PR that modifies values. Code review should include checking that new or modified values have corresponding schema entries.

Frequently Asked Questions

Does values.schema.json validate subchart values?

No. Each chart in a dependency relationship validates only its own values against its own schema. If chart A depends on chart B, and chart B has a schema, chart B’s schema validates the values under the b: key in chart A’s values.yaml — but only when processed in the context of chart B itself. Chart A’s schema should not attempt to describe chart B’s values structure. This is by design: it maintains loose coupling between charts and allows subcharts to evolve their schemas independently.

Can I use JSON Schema draft-2020-12 instead of draft-07?

Technically, Helm does not strictly enforce which draft version you use — it uses the Go library github.com/xeipuuv/gojsonschema, which supports draft-04 through draft-07. Using newer draft keywords that are not supported by this library may cause them to be silently ignored rather than throwing an error. For IDE support, draft-07 has the broadest compatibility. If you need features from newer drafts (like unevaluatedProperties from 2020-12), test carefully to confirm they are enforced by Helm’s validator and not silently skipped.

How do I handle values that differ between environments without schema conflicts?

The schema should describe all valid values across all environments. Use enum to enumerate all valid values for a field, and use if/then/else for constraints that only apply in certain configurations. The schema is a contract for what the chart accepts, not a policy for what a specific environment should use. Environment-specific policies (such as “production must use a minimum of 3 replicas”) are better enforced at a higher level — through admission controllers like OPA Gatekeeper or Kyverno — rather than in the chart schema itself.

Does schema validation run when using helm template for dry runs?

Yes. helm template runs schema validation before rendering templates. This makes it useful as a validation step in CI pipelines even without a live cluster: helm template release-name ./chart -f values-override.yaml will fail with schema errors if the values are invalid, and will output the rendered manifests if they are valid. Piping the output to kubectl apply --dry-run=client -f - adds an additional layer of Kubernetes API validation for a thorough offline check.

Should I add values.schema.json to charts I don’t maintain (upstream charts)?

For upstream charts you consume but do not maintain (such as Bitnami charts, ingress-nginx, cert-manager), the recommended approach is to maintain a separate JSON Schema file in your own GitOps repository that validates your specific values overlay files. Tools like jsonschema (Python) or ajv (Node.js) can validate a YAML/JSON values file against a schema in CI without Helm being involved. This gives you schema validation for your environment-specific overrides without needing to modify upstream chart sources.

Frequently Asked Questions

If you manage Kubernetes clusters in production, the last 18 months have been uncomfortable. Two of the most widely deployed NGINX-based Ingress Controllers have faced critical security vulnerabilities, deprecation announcements, and shifting maintenance responsibilities — all while the Kubernetes project accelerates its push toward a new traffic management standard. This is not a drill. Teams running ingress-nginx or the F5/NGINX Ingress Controller need a clear picture of what changed, what it means for their clusters, and what their realistic options are going forward.

First, Clear the Confusion: There Are Two NGINX Ingress Controllers

One of the most persistent sources of confusion in the Kubernetes networking space is that there are two completely different projects both called “NGINX Ingress Controller,” maintained by different organizations, with different architectures and different licensing.

ingress-nginx (kubernetes/ingress-nginx)

This is the community-maintained controller under the Kubernetes project umbrella, hosted at github.com/kubernetes/ingress-nginx. It uses the open-source NGINX as its data plane, configured via Lua scripting and dynamically generated nginx.conf files. This is the controller most teams end up with when they follow the official Kubernetes documentation or install from the Helm chart referenced in the ingress guide. It is free, open-source, and until recently was considered the default choice.

NGINX Ingress Controller (nginxinc/kubernetes-ingress)

This is the commercial and open-source controller maintained by F5/NGINX, hosted at github.com/nginxinc/kubernetes-ingress. It also supports NGINX Plus (the commercial version with enhanced features like active health checks, JWT authentication, and advanced load balancing). The architecture is different — it uses native NGINX APIs rather than the Lua-heavy approach — and it targets enterprise customers looking for support contracts and advanced capabilities.

These two controllers are not interchangeable. Configuration annotations differ, Helm chart values differ, and behavior under edge cases differs substantially. Understanding which one your cluster runs is the necessary starting point for any decision about migration.

# Check which NGINX IC you are actually running
kubectl get pods -n ingress-nginx -o jsonpath='{.items[*].spec.containers[*].image}'

# Community controller image looks like:
# registry.k8s.io/ingress-nginx/controller:v1.x.x

# F5/NGINX controller image looks like:
# nginx/nginx-ingress:x.x.x  or  private-registry.nginx.com/nginx-ic/nginx-plus-ingress:x.x.x

What Actually Happened: A Timeline of Disruption

The ingress-nginx CVEs (2024)

In March 2024, security researchers disclosed a set of critical vulnerabilities in ingress-nginx under the collective name IngressNightmare (CVE-2025-1097, CVE-2025-1098, CVE-2025-1974, CVE-2025-24514). The most severe of these, rated CVSS 9.8, allowed unauthenticated remote code execution against the ingress-nginx admission webhook. An attacker with network access to the admission controller could craft a malicious Ingress object to inject arbitrary NGINX configuration, ultimately achieving code execution in the controller pod — which in many clusters runs with elevated permissions and access to service account tokens across namespaces.

The vulnerabilities affected the vast majority of ingress-nginx deployments in the wild. Wiz Research, which discovered and disclosed the issues, estimated that approximately 43% of cloud environments were exposed. Patches were released in versions 1.11.5 and 1.12.1, but the incident forced uncomfortable questions about the controller’s security posture and the architecture decisions (particularly the admission webhook design) that made it possible.

Maintenance Concerns in ingress-nginx

Beyond the CVEs, the ingress-nginx project has faced ongoing concerns about maintainer bandwidth. The project is maintained by a small group of volunteers and relies heavily on community contributions. Issue response times slowed, pull requests aged, and the pace of feature development declined relative to alternatives. For a component as critical as the cluster ingress layer, this created legitimate concern about long-term sustainability without corporate backing or broader contributor growth.

F5/NGINX Deprecation Announcement

On the commercial side, F5/NGINX announced in early 2025 that the nginxinc/kubernetes-ingress controller — particularly its open-source tier — would undergo significant changes. F5 signaled a strategic shift toward NGINX Gateway Fabric, their implementation of the Kubernetes Gateway API specification. The message was clear: investment in the Ingress-based controller would be reduced, and customers were encouraged to plan migrations toward Gateway API-native solutions.

For teams running NGINX Plus-based ingress with support contracts, this was a significant business concern. The product they had licensed and standardized on was being steered toward end-of-life on the Ingress API, even if exact timelines remained somewhat ambiguous in the initial announcements.

Real Impact on Production Clusters

The practical consequences depend heavily on which controller you run and how your clusters are configured. Here is an honest assessment:

Immediate Security Risk

If you run ingress-nginx and have not patched to 1.11.5+ or 1.12.1+, your admission webhook is a critical attack surface. Patching is non-negotiable and should have happened already. The admission webhook can be disabled if you are not using it for validation (many teams are not), which significantly reduces the attack surface while you plan a longer-term migration.

# Check your current ingress-nginx version
kubectl get deployment ingress-nginx-controller -n ingress-nginx \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

# Verify admission webhook is configured
kubectl get validatingwebhookconfigurations | grep ingress

# If you need to disable the webhook temporarily (reduces but does not eliminate risk):
kubectl delete validatingwebhookconfiguration ingress-nginx-admission

Operational Uncertainty

Even after patching, the underlying questions remain. Teams are now asking: should we invest in hardening and tuning ingress-nginx knowing it may not be the strategic direction? Should we migrate now when it is our choice, rather than later when it may be forced? For NGINX IC customers, they are evaluating whether their licensing costs justify continued investment in a product being steered toward deprecation.

Configuration Migration Complexity

The real cost of migration is in the annotation-heavy configurations that accumulate over time. Teams that have built complex routing logic using nginx.ingress.kubernetes.io/* annotations — custom headers, rate limiting, auth snippets, rewrite rules, canary traffic splitting — face significant rework when switching controllers. This is the primary reason many teams are reluctant to move despite clear signals that a transition is coming.

The Alternatives: An Honest Evaluation

There is no shortage of Ingress controller options. The question is which alternatives are mature enough for production workloads at scale, and what trade-offs each brings.

Traefik

Traefik Proxy (and its Kubernetes-native version via Traefik Hub) has emerged as the most popular alternative for teams leaving ingress-nginx. It supports the standard Kubernetes Ingress API for drop-in compatibility, its own IngressRoute CRDs for advanced features, and Kubernetes Gateway API. It is written in Go, has strong TLS automation via Let’s Encrypt, and has excellent observability with built-in metrics and a real-time dashboard.

Trade-offs: Traefik’s configuration model is different enough from NGINX that complex routing logic requires rethinking rather than translating. Performance under very high connection counts is generally good but NGINX has a longer track record in extreme-scale deployments. The commercial offering (Traefik Hub) adds API gateway capabilities but introduces vendor dependency.

Envoy Gateway

Envoy Gateway is now a CNCF project and implements the Kubernetes Gateway API natively using Envoy as its data plane. This is arguably the most strategically aligned option for teams that want to bet on the future of Kubernetes networking. Envoy is battle-tested (it powers Istio, Contour, and large-scale service meshes at companies like Lyft and Google), and the Gateway API implementation is comprehensive and actively developed.

Trade-offs: Envoy Gateway is relatively young as a standalone project. Teams unfamiliar with Envoy will face a steeper learning curve for debugging and custom configuration. The operational model differs significantly from NGINX-based controllers. However, for greenfield deployments or teams willing to invest in the transition, this is a strong forward-looking choice.

Cilium Gateway API

If your cluster already runs Cilium as the CNI, enabling Gateway API support is a natural evolution. Cilium’s Gateway API implementation leverages eBPF for high-performance packet processing, avoiding the overhead of userspace proxy hops entirely. It is deeply integrated with Cilium’s network policy model and observability stack (Hubble).

Trade-offs: This option is only relevant if you are already committed to Cilium as your CNI, or are willing to make that switch simultaneously. Migrating both the CNI and the ingress layer at the same time is a significant operational risk. For Cilium shops, however, this consolidates complexity and provides excellent performance and observability.

HAProxy Ingress

HAProxy Ingress Controller is maintained by the HAProxy Technologies team and has a strong reputation for raw performance and precise traffic control. It supports both Ingress and Gateway API and has a long track record in high-throughput production environments. For teams with existing HAProxy expertise, it provides a familiar mental model for load balancing configuration.

Trade-offs: Smaller community than Traefik or NGINX. Less ecosystem tooling and fewer tutorials. Best suited for teams that specifically want HAProxy’s capabilities (fine-grained connection management, advanced health checking, TCP/HTTP mode flexibility) rather than as a default choice.

Kong Ingress Controller

Kong bridges the gap between an Ingress controller and a full API gateway. It supports Ingress and Gateway API resources alongside its own Kong-native plugin system for authentication, rate limiting, transformation, and observability. For teams that need API gateway capabilities rather than pure L7 routing, Kong provides a unified platform.

Trade-offs: Kong adds operational complexity. Running Kong requires either a PostgreSQL database (DB-mode) or careful management of declarative configuration (DB-less mode). The plugin ecosystem is powerful but introduces additional configuration surface. For teams that just need ingress routing, Kong may be more than necessary. For teams building API platforms, it is worth the overhead.

Istio Gateway

Istio’s ingress gateway (now aligned with Gateway API via its Kubernetes Gateway API integration) provides entry-point traffic management as part of a full service mesh. If your organization is planning or running Istio for east-west traffic, using Istio’s gateway for north-south traffic creates a unified data plane (Envoy) and consistent observability across all service communication.

Trade-offs: Istio is a serious operational commitment. The control plane overhead, the learning curve, and the impact on pod scheduling and sidecar management are significant. Choosing Istio purely for ingress replacement is like buying a race car because you needed a vehicle with good brakes. Consider this path only if service mesh capabilities are on your roadmap.

NGINX Gateway Fabric (F5’s Gateway API implementation)

F5/NGINX is building NGINX Gateway Fabric as their strategic forward path — an NGINX-based implementation of the Kubernetes Gateway API. For teams heavily invested in NGINX and wanting to stay in that ecosystem while moving to Gateway API, this provides a migration path within familiar territory. It is still maturing but represents where F5 is putting its development resources.

Comparison Matrix

Gateway API: The Strategic Direction You Cannot Ignore

The Kubernetes Gateway API is not simply “Ingress v2.” It is a fundamentally richer traffic management model designed to address the limitations that drove teams to annotation-based workarounds for the past several years. Understanding it is essential regardless of which controller you choose, because the ecosystem is clearly converging on it.

The core resource hierarchy consists of GatewayClass (defines a type of gateway, created by infrastructure providers), Gateway (a specific instance of a listener configuration, typically managed by platform teams), and HTTPRoute, TCPRoute, GRPCRoute, and other route resources (managed by application teams). This separation of concerns maps cleanly onto organizational roles — infrastructure teams control the gateway, application teams control their routing rules.

# Example Gateway API resources replacing an ingress-nginx Ingress
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production-gateway
  namespace: infra
spec:
  gatewayClassName: nginx  # or envoy, traefik, cilium, etc.
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
      - name: wildcard-tls
    allowedRoutes:
      namespaces:
        from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app
  namespace: my-app-namespace
spec:
  parentRefs:
  - name: production-gateway
    namespace: infra
  hostnames:
  - "app.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: my-api-service
      port: 8080
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: my-frontend-service
      port: 3000

Gateway API reached v1.0 (GA for HTTPRoute and Gateway) in October 2023, and v1.1 followed in 2024 with GRPCRoute graduation and expanded features. The project has broad support across controllers (Traefik, Envoy Gateway, Cilium, NGINX Gateway Fabric, Kong, Istio, and others all implement it). The Ingress API is not being removed from Kubernetes, but new feature development is effectively frozen — Gateway API is where capabilities like traffic weighting, header manipulation, request mirroring, and backend protocol configuration are being built.

Decision Framework: Stay, Migrate, or Evaluate?

There is no universal right answer. The following framework helps teams make a context-appropriate decision rather than following hype or panic.

Stay on ingress-nginx if:

• You have patched to 1.11.5+ or 1.12.1+ and have disabled or hardened the admission webhook
• Your cluster is stable, heavily annotation-dependent, and migration cost outweighs risk
• You have internal NGINX expertise and can take ownership of monitoring the project’s maintenance health
• Your organization has a short-term horizon (decommissioning or major platform change within 12-18 months)

Migrate now if:

• You are running the F5/NGINX IC with a support contract that is being deprecated
• Your cluster has moderate annotation complexity and you have engineering cycles available
• You are planning a major Kubernetes version upgrade or cluster rebuild — do it at the same time
• Your security team has flagged the CVE history as unacceptable for your risk profile
• You are building a new cluster or platform team and want to standardize on Gateway API from the start

Evaluate before committing if:

• Your workloads have complex traffic requirements (WebSockets, gRPC, canary deployments, header-based routing) that differ significantly across controllers
• You are considering Gateway API but the specific controllers in your environment have not graduated their Gateway API implementations yet
• You have multi-cluster or multi-tenant requirements that change the analysis
• You need to assess total cost including commercial support, tooling changes, and team retraining

Migration Checklist

For teams that have decided to migrate, the following sequence reduces risk and ensures nothing critical is missed:

Phase 1: Inventory and Assessment

• Enumerate all Ingress resources across all namespaces and document their annotations
• Identify annotations with no direct equivalent in your target controller
• Map TLS certificate sources (cert-manager, Secrets, external providers) and confirm compatibility
• Document any custom NGINX configuration snippets (nginx.ingress.kubernetes.io/configuration-snippet, server-snippet) — these are high-risk items that require manual translation
• Inventory any rate limiting, authentication, or WAF configurations layered on the controller

# Enumerate all ingress resources and their annotations across the cluster
kubectl get ingress -A -o json | jq -r '
  .items[] |
  {
    namespace: .metadata.namespace,
    name: .metadata.name,
    annotations: (.metadata.annotations // {} | keys)
  }
'

Phase 2: Target Controller Validation

• Deploy target controller to a non-production cluster with identical Ingress/HTTPRoute resources
• Validate TLS termination, redirect behavior, and timeout configurations
• Run load tests to confirm performance characteristics match expectations
• Validate observability — metrics, logs, and traces integrate with your existing stack
• Test failure scenarios: backend unavailability, certificate expiry, controller pod restart

Phase 3: Staged Production Migration

• Deploy new controller to production alongside existing controller (different IngressClass)
• Migrate low-risk, low-traffic Ingress resources first by updating their ingressClassName
• Use DNS-based canary switching (weighted routing at the DNS level) rather than switching entire IngressClass at once
• Monitor error rates and latency for 24-48 hours after each batch migration
• Migrate critical services during low-traffic windows with rollback plan documented
• Decommission old controller only after all resources are migrated and validated

# Migrate individual Ingress to new controller by changing ingressClassName
kubectl patch ingress my-app -n my-namespace \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/ingressClassName", "value": "traefik"}]'

# Or if migrating to Gateway API, create equivalent HTTPRoute first,
# test it, then remove the old Ingress resource
kubectl apply -f my-app-httproute.yaml
# Validate, then:
kubectl delete ingress my-app -n my-namespace

Phase 4: Gateway API Adoption (Optional but Recommended)

• Install Gateway API CRDs if not already present (kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml)
• Define GatewayClass resources matching your chosen controller
• Migrate Ingress resources to HTTPRoute progressively, starting with simpler configurations
• Update CI/CD pipelines and Helm charts to generate HTTPRoute instead of Ingress resources for new services
• Establish a policy: new services use Gateway API; legacy services migrate on their next significant update

Recommendation

For most platform engineering teams reading this in 2025, the pragmatic recommendation is as follows:

Short term (next 30 days): Patch ingress-nginx to the latest release if you are still on it. Assess and harden or disable the admission webhook. This is not optional.

Medium term (3-6 months): Evaluate Traefik or Envoy Gateway against your specific workload requirements. Traefik is the lower-friction migration for teams coming from ingress-nginx on the Ingress API. Envoy Gateway is the stronger strategic choice if you are willing to commit to Gateway API fully. Either way, run a parallel deployment in a non-production environment and measure the delta in operational overhead.

Long term (6-18 months): Plan migration to Gateway API resources regardless of which data plane you choose. The Ingress API will not disappear overnight, but feature parity with Gateway API capabilities will never arrive. Teams that standardize on Gateway API now build the institutional knowledge that will be valuable as the ecosystem continues to evolve.

If you are running F5/NGINX IC under a support contract: engage your F5 account team now to get a clear timeline on the deprecation path and evaluate NGINX Gateway Fabric as a within-ecosystem migration before looking at alternatives. The question is not whether to migrate but when and to what.

Avoid the temptation to treat this as a purely technical decision. The switch of an ingress controller touches CI/CD pipelines, monitoring dashboards, runbooks, on-call playbooks, and engineering team knowledge. Factor in the total transition cost, not just the YAML changes.

Frequently Asked Questions