Kubernetes Security Best Practices: 2026 Production Hardening Guide

Kubernetes Security Best Practices: 2026 Production Hardening Guide

Kubernetes security is not a single feature you enable — it is a layered discipline that spans the control plane, workloads, networking, supply chain, and runtime. Get one layer wrong and the others rarely save you. This guide covers the controls that matter most in production, why each one exists, and how to implement them without breaking your cluster — plus a prioritized roadmap so you know what to do in your first week, not just an undifferentiated list of “best practices.”

Related reading: an immutable OS like Talos Linux.

Related reading: the authentication-inside-containers anti-pattern.

Let me start with the part most hardening guides skip: what an actual attack looks like.

Anatomy of a Real Kubernetes Attack Chain

Abstract advice (“apply least privilege”) doesn’t land until you’ve seen how a single misconfiguration cascades. Here is a realistic chain — every step maps to a documented technique in the MITRE ATT&CK for Containers matrix. If you haven’t seen it before, ATT&CK is an industry-standard, openly maintained knowledge base of real-world adversary behavior: a catalogue of how attackers actually operate, organized by goal (initial access, credential access, lateral movement, and so on). It’s the common language security teams use to describe and defend against attacks.

  1. Initial access. An application pod runs a vulnerable image — say, an unpatched dependency with a remote code execution (RCE) flaw, a bug that lets an attacker run arbitrary code on the host process. The attacker gets code execution inside the container. So far, container isolation should contain the blast radius.
  2. Credential access. The pod has automountServiceAccountToken: true (the default). The attacker reads /var/run/secrets/kubernetes.io/serviceaccount/token — a valid API credential, handed to them for free.
  3. Discovery. Using that token, the attacker queries the API server. The ServiceAccount was bound to a convenient cluster-admin role “to unblock a deploy.” Now they can list every Secret in every namespace.
  4. Lateral movement. They read database credentials, cloud provider keys, and other ServiceAccount tokens from Secrets. The flat pod network (no NetworkPolicies) lets them reach internal services directly.
  5. Privilege escalation / escape. They schedule a privileged pod with hostPID and the host filesystem mounted, then break out to the node. From the node, they reach the kubelet and other tenants’ workloads.
  6. Impact. Crypto-mining, data exfiltration, or ransomware across the cluster.

Notice that steps 2 through 5 each had a one-line fix: disable token automount, scope the RBAC, encrypt Secrets / use an external store, apply default-deny NetworkPolicies, enforce Pod Security. Defense in depth means an attacker has to defeat every layer — and most attackers give up when the easy chain breaks. The rest of this guide is those layers, ordered by how much they shrink that chain.

The Kubernetes Attack Surface

Before hardening anything, understand what you are protecting. A Kubernetes cluster has several distinct attack surfaces:

  • API server — The central control plane. Any entity that can reach it with valid credentials can read cluster state, modify workloads, or escalate privileges.
  • etcd — Stores all cluster state in plain text, including Secrets. Direct etcd access is equivalent to root on every node.
  • Nodes — A compromised node can access all Secrets mounted on pods running on it, access the kubelet API, and potentially escape to the hypervisor.
  • Pods — Privileged pods, host-network pods, and pods with excessive capabilities can break container isolation.
  • Supply chain — Malicious images, compromised registries, and unsigned artifacts can introduce attacker-controlled code into your cluster.
  • RBAC — Overly permissive roles allow lateral movement and privilege escalation once an attacker gains any foothold.

Prioritize based on your threat model — a public-facing multi-tenant cluster needs all of these; an internal development cluster can relax some.

The First-Week Hardening Roadmap (Prioritized)

If you inherited a cluster with nothing in place, do not try to do everything at once. Order matters — some controls give huge risk reduction for minimal effort and zero breakage risk, others need careful rollout. This is the sequence I use:

DayControlRisk reductionBreakage risk
1Audit RBAC, remove stray cluster-admin, disable unused SA token automountHighLow
1Enable API server audit loggingMedium (visibility)None
2Pod Security Admission in warn + audit mode (all namespaces)HighNone (warn only)
3Deploy image scanning in CI (Trivy/Grype), fail on CriticalHighLow
4NetworkPolicies in audit-style rollout: default-deny in one namespace firstHighMedium — test DNS!
5Enable etcd encryption at rest / move Secrets to external storeHighLow
6Flip Pod Security Admission to enforce: baseline, then restricted per namespaceHighMedium
7Deploy runtime detection (Falco) + continuous scanning (Trivy Operator)MediumNone

The single most important idea: roll out enforcing controls in observation mode first (warn/audit for Pod Security, default-deny NetworkPolicies in one namespace). You want to discover what breaks in a dashboard, not in an incident.

Tools to automate and report each step

You don’t have to do any of this by hand. Each step has tooling that both applies the control and reports on its state, so you can wire it into CI or a recurring job:

1. RBAC: Least Privilege from Day One

Role-Based Access Control is Kubernetes’ primary authorization mechanism. Most clusters fail at RBAC not because it is misconfigured, but because it is over-permissive by default and nobody reviews it systematically.

Common RBAC Mistakes

  • Binding to cluster-admin for convenience. Almost no workload needs cluster-admin. Use namespaced roles wherever possible.
  • Using * verbs or resources in roles. Wildcard permissions are almost always broader than intended.
  • Not auditing ServiceAccount token usage. Every pod gets a ServiceAccount. Custom workloads often get over-permissive SAs.
  • Forgetting automountServiceAccountToken: false. If a workload does not need to talk to the Kubernetes API, disable token mounting entirely — this single setting breaks step 2 of the attack chain above.

Practical RBAC Patterns

For a workload that only needs to read ConfigMaps in its own namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: configmap-reader
  namespace: my-app
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-app-configmap-reader
  namespace: my-app
subjects:
- kind: ServiceAccount
  name: my-app
  namespace: my-app
roleRef:
  kind: Role
  name: configmap-reader
  apiGroup: rbac.authorization.k8s.io

And disable token automount on the workload that doesn’t call the API at all:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: my-app
automountServiceAccountToken: false

Audit existing RBAC with kubectl-who-can or rbac-tool to find overly permissive bindings before attackers do. A useful one-liner: list every subject that can read Secrets cluster-wide with kubectl who-can get secrets.

2. Pod Security Standards (and Migrating off PodSecurityPolicy)

PodSecurityPolicy was deprecated in Kubernetes 1.21 and removed in 1.25. Its replacement is Pod Security Admission (PSA), a built-in admission controller that enforces one of three Pod Security Standards profiles at the namespace level:

  • Privileged — No restrictions. For system components only.
  • Baseline — Prevents the most critical privilege escalations: privileged containers, hostPID, hostIPC, hostNetwork, dangerous capabilities.
  • Restricted — Enforces current hardening best practices. Requires running as non-root, dropping all capabilities, and using a restricted seccomp profile.

Enable enforcement at the namespace level with labels:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.30
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.30
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.30

A pod that runs as root or requests host-network in a namespace enforcing restricted will be rejected at admission. The warn and audit modes let you test before enforcing. For a full walkthrough of how PSA evaluates pods and how to roll it out, see my guide on understanding Pod Security Admission.

Migrating from PodSecurityPolicy to PSA

If you’re still on a cluster that used PSP, the migration path is:

  1. Map your PSPs to the closest PSA level. Most “restricted” PSPs map to restricted; permissive ones to baseline. The official pspmigrator tool can suggest mappings.
  2. Label every namespace in warn/audit mode matching that level — no enforcement yet.
  3. Watch the audit logs and warnings for a release cycle. Fix the workloads that would be rejected (add securityContext, drop capabilities).
  4. Flip to enforce namespace by namespace, starting with the least critical.

PSA is intentionally coarse-grained — three levels, namespace-scoped. For anything finer (per-team registries, required labels, custom mutation), you need a policy engine, which is the next section.

3. Policy Engines: Kyverno vs OPA Gatekeeper

Once you outgrow PSA’s three levels, you need an admission policy engine. The two standards are Kyverno and OPA Gatekeeper, and choosing between them is one of the most common platform decisions.

KyvernoOPA Gatekeeper
Policy languageYAML (Kubernetes-native)Rego (purpose-built DSL)
Learning curveLow — looks like other manifestsSteep — Rego is its own paradigm
Mutation supportYes, first-classLimited
Image verification (Cosign)Built-inVia external data
Best whenTeam wants fast adoption, K8s-onlyTeam already runs OPA across the stack

For most teams without existing Rego expertise, Kyverno is significantly faster to adopt and maintain. A Kyverno policy to require all images come from your private registry:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-registries
    match:
      any:
      - resources:
          kinds: ["Pod"]
    validate:
      message: "Images must come from registry.company.com"
      pattern:
        spec:
          containers:
          - image: "registry.company.com/*"

Both integrate cleanly with GitOps — store policies in Git, apply via Argo CD or Flux, and you get an auditable history of every policy change. I’ve written several deep dives on this: Kyverno: enforcing standard and custom policies, extending Kyverno with custom rules, and running the Kyverno CLI in CI/CD with GitHub Actions — or browse everything under the policies tag.

4. Network Policies: Micro-Segmentation

By default, every pod in a Kubernetes cluster can communicate with every other pod across all namespaces. This flat network model gives attackers unrestricted lateral movement once they compromise any workload (step 4 of the attack chain).

Network Policies define L3/L4 allow-rules for pod-to-pod communication. They are enforced by your CNI (Container Network Interface) plugin (Calico, Cilium, Weave — not Flannel, which does not support NetworkPolicy).

Default Deny Pattern

Start by denying all ingress and egress in a namespace, then open only what is explicitly needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Then allow specific traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - protocol: TCP
      port: 5432

The DNS Trap (the #1 reason default-deny “breaks everything”)

The most common NetworkPolicy support ticket: “I applied default-deny and the whole namespace stopped working.” The cause is almost always DNS. A default-deny egress policy blocks the pod from reaching kube-dns, so every name resolution fails and applications appear to hang or crash-loop.

Always pair default-deny egress with an explicit DNS allow rule:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Roll default-deny out in one namespace first, confirm DNS and required egress work, then expand. Tools like Cilium’s Hubble or Calico’s flow logs make it much easier to see exactly which flows you need to allow.

5. Secrets Management

Kubernetes Secrets are base64-encoded, not encrypted. They are stored in etcd in plain text by default. Anyone with get permission on Secrets can read them. This is not a vulnerability — it is a design decision that puts the responsibility on you to:

  • Enable encryption at rest for etcd. Configure EncryptionConfiguration with an AES-CBC or AES-GCM provider so Secrets are encrypted before being written to etcd.
  • Use external secret stores. HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault with the External Secrets Operator means actual secret values never live in Kubernetes at all.
  • Restrict Secret RBAC aggressively. Never give list on Secrets cluster-wide — it returns all values. Use get on named resources where possible.
  • Avoid environment variables for secrets. Prefer volume mounts. Env vars are visible in pod inspect output and can leak through application logging.
# etcd encryption at rest - in kube-apiserver config
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-32-byte-key>
  - identity: {}

For the full external-store pattern, see the guide on injecting secrets into pods with HashiCorp Vault.

6. Image Security and Supply Chain

Your runtime security posture is only as good as the images you run. A compromised image from a public registry bypasses every runtime control you have.

Scan images in CI

Use Trivy, Grype, or Snyk to scan images as part of CI. Block deployments of images with critical CVEs (Common Vulnerabilities and Exposures — publicly catalogued security flaws). I’ve covered the practical side of this in scanning Docker images with Trivy, scanning your images locally before they ship, and a broader roundup of open-source development security tools.

# In your CI pipeline
trivy image --exit-code 1 --severity CRITICAL your-image:tag

Use a private registry with admission control

Only allow images from your private registry using an admission webhook (Kyverno, OPA Gatekeeper) — the policy in section 3 does exactly this. It prevents developers from running arbitrary public images in production.

Use distroless or minimal base images

Distroless images contain only the application and its runtime dependencies — no shell, no package manager, no debugging tools. This drastically reduces both the attack surface and the CVE count. Google’s distroless images are available for Java, Node.js, Python, and Go. (Related: debugging distroless containers when you do need to inspect one.)

Sign and verify images (and the SLSA angle)

Cosign (from the Sigstore project) lets you sign container images and verify signatures at admission time using Kyverno or Connaisseur. This prevents image-substitution attacks where an attacker replaces a legitimate image in your registry.

If you’re being asked about supply-chain compliance, the framework to know is SLSA (Supply-chain Levels for Software Artifacts). The practical progression: SLSA L1 = you have a build provenance document; L2 = it’s signed and the build is hosted; L3 = the build is hardened and non-falsifiable. Generating provenance with your CI (GitHub Actions has native SLSA generators) and verifying it at admission with Cosign + Kyverno gets you most of the way to L2/L3 without a platform rebuild.

7. Runtime Security

Runtime security detects and responds to malicious activity after a container is running. The primary tool is Falco — a CNCF project that uses eBPF (extended Berkeley Packet Filter — a Linux kernel technology for running sandboxed observability programs) to monitor system calls and raise alerts when containers behave unexpectedly.

Default Falco rules catch common attack patterns:

  • Shell spawned in a container
  • Network connection to an unexpected IP
  • Write to a sensitive file path (/etc/passwd, /etc/shadow)
  • Privilege escalation via setuid binaries
  • Container drift (new executable files written at runtime)

Combine Falco with seccomp profiles to restrict the system calls a container can make at the kernel level. The RuntimeDefault seccomp profile (a default option since Kubernetes 1.27) blocks 300+ system calls that containers virtually never need.

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 65534
      capabilities:
        drop: ["ALL"]

These four securityContext settings together (allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, runAsNonRoot: true, capabilities.drop: ALL) make container escape significantly harder and satisfy the Kubernetes Restricted pod security standard. They directly close step 5 of the attack chain.

8. API Server Hardening

The API server is the most critical component to harden. Key settings:

  • Disable anonymous authentication. --anonymous-auth=false ensures every request is authenticated.
  • Enable audit logging. Log all API server requests to a file or webhook. Without audit logs, you cannot investigate incidents or detect RBAC abuse.
  • Restrict admission plugins. Ensure NodeRestriction is enabled — it prevents node kubelets from modifying objects outside their own node.
  • Do not expose the API server to the internet. Use a VPN, bastion host, or private endpoint. If you must expose it, restrict access by IP.
# Minimal audit policy - log all requests at metadata level,
# and full request body for sensitive resources
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets", "configmaps"]
- level: Metadata
  omitStages: ["RequestReceived"]

9. etcd Security

etcd stores all cluster state. Treat it as sensitive as your production database:

  • Enable TLS for all etcd communication — both peer (etcd-to-etcd) and client (apiserver-to-etcd) with mutual TLS.
  • Restrict network access to etcd. It should only be reachable by the API server. Use firewall rules or security groups.
  • Enable encryption at rest (see Secrets section).
  • Back up etcd regularly. A snapshot is a complete copy of all cluster state, including all Secrets. Encrypt backups and store them separately from the cluster.

10. Multi-Tenancy Isolation

If multiple teams or customers share a cluster, namespace boundaries alone are not a security boundary — they’re an organizational one. Hardening multi-tenant clusters adds requirements on top of everything above:

  • Namespace-per-tenant with ResourceQuotas and LimitRanges to prevent noisy-neighbor and resource-exhaustion DoS.
  • NetworkPolicies that deny cross-namespace traffic by default, so tenant A cannot reach tenant B’s pods.
  • A policy engine enforcing per-tenant rules (allowed registries, required labels, no hostPath).
  • Separate node pools for untrusted workloads, or a sandboxed runtime (gVisor, Kata Containers) when you run genuinely untrusted code.

For hard multi-tenancy (untrusted tenants), the honest answer is that vanilla namespaces aren’t enough — consider virtual clusters (vCluster) or separate clusters entirely. Soft multi-tenancy (trusted internal teams) is well served by the controls in this guide.

11. Benchmarks and Continuous Posture

CIS Kubernetes Benchmark

The CIS Kubernetes Benchmark is a comprehensive checklist covering the control plane, nodes, and workloads. Running kube-bench gives you a scored assessment:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs $(kubectl get pods -l app=kube-bench -o name)

kube-bench outputs PASS/FAIL/WARN for each control with remediation guidance. Run it after initial cluster setup and after major configuration changes.

Continuous scanning with Trivy Operator / Kubescape

Kubescape and the Trivy Operator provide continuous security scanning of live cluster state — not just a one-time audit. They check workloads against NSA/CISA hardening guidelines, the MITRE ATT&CK framework, and the CIS benchmark in real time.

helm repo add aquasecurity https://aquasecurity.github.io/helm-charts/
helm install trivy-operator aquasecurity/trivy-operator 
  --namespace trivy-system 
  --create-namespace 
  --set="trivy.ignoreUnfixed=true"

Trivy Operator creates VulnerabilityReport, ConfigAuditReport, and RbacAssessmentReport custom resources alongside each workload. Scrape them with Prometheus and build a security dashboard in Grafana.

Security Hardening Checklist

  • ✅ RBAC reviewed — no wildcard roles, no unnecessary cluster-admin bindings
  • ✅ ServiceAccount token automount disabled for workloads that do not need API access
  • ✅ Pod Security Standards enforced at namespace level (at least Baseline, Restricted where possible)
  • ✅ Policy engine (Kyverno/Gatekeeper) enforcing registry, label, and mutation rules
  • ✅ Network policies deployed — default deny with explicit allows (including DNS!)
  • ✅ Secrets encrypted at rest in etcd or moved to an external store
  • ✅ Images scanned in CI — no critical CVEs in production
  • ✅ Private registry enforced via admission control
  • ✅ Image signing + verification (Cosign) and build provenance (SLSA)
  • ✅ Container securityContext hardened (non-root, read-only fs, no capabilities)
  • ✅ seccomp RuntimeDefault profile enabled
  • ✅ API server audit logging enabled, anonymous auth disabled
  • ✅ etcd TLS and network access restricted
  • ✅ Multi-tenancy isolation (quotas, cross-namespace deny) if shared
  • ✅ kube-bench run and critical/high findings remediated
  • ✅ Runtime security (Falco) deployed and alerts routed to on-call
  • ✅ Continuous scanning (Trivy Operator or Kubescape) deployed

FAQ

Where do I start if my cluster has no security controls today?

Follow the first-week roadmap above. The short version: audit RBAC (revoke stray cluster-admin), enable Pod Security Admission in warn mode on all namespaces, and deploy image scanning + Trivy Operator. These give immediate visibility and stop the most common privilege escalations without breaking anything.

Does enabling Network Policies break DNS resolution?

Yes — this is the single most common failure. A default-deny egress policy blocks pods from reaching kube-dns, so name resolution fails. Add an egress rule allowing UDP and TCP port 53 to the kube-system namespace whenever you apply default-deny (see the DNS allow policy above).

Should I use OPA Gatekeeper or Kyverno?

Both enforce admission policies. Kyverno is Kubernetes-native (policies are YAML) while Gatekeeper uses Rego. For teams without Rego expertise, Kyverno is faster to adopt and supports mutation and Cosign verification out of the box. Choose Gatekeeper if you already run OPA elsewhere and want one policy language across your stack.

What replaced PodSecurityPolicy?

Pod Security Admission (PSA), built into Kubernetes since 1.25. It enforces three profiles (privileged/baseline/restricted) via namespace labels. For finer-grained control than PSA’s three levels, add Kyverno or Gatekeeper.

Is Kubernetes certified for PCI-DSS or SOC 2?

Kubernetes itself is not certified — your configuration and the controls you implement determine compliance. The CIS Kubernetes Benchmark maps to many PCI-DSS and SOC 2 requirements. Managed offerings (EKS, GKE, AKS) carry their own compliance certifications for the underlying infrastructure.

How often should I update Kubernetes for security patches?

Apply a patch release within 30 days for High/Critical CVEs. Minor version upgrades (e.g., 1.30 → 1.31) should happen within the support window — Kubernetes maintains the last three minor versions. Falling more than one minor behind means running without patches for a growing subset of the codebase.

Are namespaces a security boundary?

No. Namespaces are an organizational boundary. Real isolation between tenants requires NetworkPolicies, ResourceQuotas, a policy engine, and — for untrusted workloads — sandboxed runtimes (gVisor/Kata) or separate clusters.


For a deeper look at how security fits into the broader Kubernetes platform architecture, see the Kubernetes architecture patterns guide and the guide on building a security-first Kubernetes culture.

ArgoCD Guide: GitOps Continuous Delivery for Kubernetes

ArgoCD Guide: GitOps Continuous Delivery for Kubernetes

ArgoCD has become the de facto standard for GitOps-based continuous delivery in Kubernetes. If you are running production workloads on Kubernetes and still deploying with raw kubectl apply or untracked Helm releases, ArgoCD solves a class of problems you may not even know you have yet. This guide covers everything from core concepts to production-grade configuration.

The Problem ArgoCD Solves

Traditional CI/CD pushes deployments into a cluster. A CI system runs tests, builds an image, and then executes kubectl apply or helm upgrade against the cluster. This model has several structural problems:

  • Drift goes undetected. Someone applies a hotfix directly to the cluster. Now your Git repository no longer reflects reality, and nobody knows it.
  • No single source of truth. The cluster state is authoritative, not Git. Your desired state and actual state can diverge silently.
  • Rollback is painful. Rolling back a bad deployment means re-running old CI pipelines or manually reversing changes, neither of which is fast.
  • Multi-cluster management compounds the problem. Each cluster becomes a snowflake with its own history of undocumented changes.

GitOps inverts this model. Git is the source of truth. The cluster pulls its desired state from Git and continuously reconciles toward it. ArgoCD is the most mature GitOps operator for Kubernetes, implementing this pull-based model with a production-ready feature set.

How ArgoCD Works: Core Architecture

ArgoCD runs as a set of controllers inside your Kubernetes cluster. The core components are:

  • Application Controller — Watches both the Git repository and the live cluster state. Computes the diff and drives reconciliation.
  • API Server — Exposes the gRPC/REST API consumed by the CLI, UI, and external systems.
  • Repository Server — Generates Kubernetes manifests from source (Helm, Kustomize, plain YAML, Jsonnet).
  • Redis — Caches cluster state and repository data to reduce API server load.
  • Dex (optional) — Provides OIDC authentication for SSO integration.

The fundamental unit in ArgoCD is an Application — a CRD that maps a source (a path in a Git repo at a specific revision) to a destination (a namespace in a cluster). ArgoCD continuously compares the desired state from Git with the live state in the cluster and reports on the sync status.

Sync Status vs Health Status

Two orthogonal concepts you need to understand from day one:

  • Sync Status — Does the live state match what Git says it should be? Values: Synced, OutOfSync, Unknown.
  • Health Status — Is the application actually working? Values: Healthy, Progressing, Degraded, Suspended, Missing, Unknown.

An application can be Synced but Degraded — the manifests were applied correctly, but a pod is crash-looping. Conversely, it can be OutOfSync but Healthy — someone applied a change directly to the cluster outside of Git.

Installing ArgoCD

The official installation method uses a single manifest. For production, always pin to a specific version:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.11.0/manifests/install.yaml

This deploys ArgoCD in the argocd namespace with full cluster-admin access. For a production HA setup, use the manifests/ha/install.yaml variant, which deploys multiple replicas of the API server and application controller.

Accessing the UI and CLI

The initial admin password is auto-generated and stored in a secret:

argocd admin initial-password -n argocd

For local access, port-forward the API server:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Then log in via the CLI:

argocd login localhost:8080 --username admin --password <password> --insecure

For production, expose the ArgoCD server via an Ingress or LoadBalancer with a proper TLS certificate. If you’re using NGINX Ingress Controller:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  rules:
  - host: argocd.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              number: 443

Defining Your First Application

Applications can be created via the UI, the CLI, or declaratively with a YAML manifest. The declarative approach is the recommended one — it means your ArgoCD configuration itself is in Git:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/your-app
    targetRevision: HEAD
    path: k8s/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Key fields to understand:

  • targetRevision — Can be a branch name, tag, or commit SHA. For production, pin to a tag rather than HEAD.
  • path — The directory within the repo containing your Kubernetes manifests.
  • automated.prune — Automatically delete resources that are no longer in Git. Required for true GitOps but use carefully — it will delete things.
  • automated.selfHeal — Automatically revert manual changes made directly to the cluster. This is what enforces Git as the single source of truth.

Helm Integration

ArgoCD has native Helm support. It can deploy Helm charts directly from chart repositories or from your Git repository. You can override values per environment:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prometheus-stack
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    chart: kube-prometheus-stack
    targetRevision: 58.4.0
    helm:
      releaseName: prometheus-stack
      valuesObject:
        grafana:
          adminPassword: "${GRAFANA_PASSWORD}"
        prometheus:
          prometheusSpec:
            retention: 30d
            storageSpec:
              volumeClaimTemplate:
                spec:
                  storageClassName: fast-ssd
                  resources:
                    requests:
                      storage: 50Gi
  destination:
    server: https://kubernetes.default.svc
    namespace: observability

One important nuance: ArgoCD renders Helm charts server-side using its own templating engine, not helm install. This means Helm hooks (pre-install, post-upgrade, etc.) are supported, but the release is not tracked in Helm’s release history. Running helm list will not show ArgoCD-managed releases unless you configure ArgoCD to use the Helm secrets backend.

Projects: Multi-Tenancy and Access Control

ArgoCD Projects provide multi-tenancy within a single ArgoCD instance. They let you restrict which source repositories, destination clusters, and namespaces a team can deploy to. Every Application belongs to a Project.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform-team
  namespace: argocd
spec:
  description: Platform team applications
  sourceRepos:
  - 'https://github.com/your-org/*'
  destinations:
  - namespace: 'platform-*'
    server: https://kubernetes.default.svc
  clusterResourceWhitelist:
  - group: ''
    kind: Namespace
  namespaceResourceBlacklist:
  - group: ''
    kind: ResourceQuota

Projects are where you define the boundaries of what each team can do. The default project has no restrictions — never use it for production workloads. Create dedicated projects per team or per environment.

RBAC Configuration

ArgoCD has its own RBAC system layered on top of Kubernetes RBAC. It is configured via the argocd-rbac-cm ConfigMap. Roles are defined per project or globally:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Platform team has full access to platform-team project
    p, role:platform-admin, applications, *, platform-team/*, allow
    p, role:platform-admin, projects, get, platform-team, allow
    p, role:platform-admin, repositories, *, *, allow

    # Dev team can sync but not delete
    p, role:developer, applications, get, */*, allow
    p, role:developer, applications, sync, */*, allow
    p, role:developer, applications, action/*, */*, allow

    # Bind SSO groups to roles
    g, your-org:platform-team, role:platform-admin
    g, your-org:developers, role:developer

The policy.default: role:readonly ensures that any authenticated user who has no explicit role assignment gets read-only access — a safe default for production.

Multi-Cluster Management

ArgoCD can manage multiple Kubernetes clusters from a single control plane. Register external clusters with the CLI:

# First, ensure the target cluster context is in your kubeconfig
argocd cluster add production-eu-west --name production-eu-west

# Verify registration
argocd cluster list

ArgoCD will create a ServiceAccount in the target cluster and store its credentials as a Kubernetes secret in the ArgoCD namespace. Applications can then target this cluster by name in their destination.server field.

For large-scale multi-cluster setups, consider the App of Apps pattern or ApplicationSets. ApplicationSets are a controller that generates Applications dynamically based on generators — cluster lists, Git directory structures, or matrix combinations:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-addons
  namespace: argocd
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          environment: production
  template:
    metadata:
      name: '{{name}}-addons'
    spec:
      project: platform
      source:
        repoURL: https://github.com/your-org/cluster-addons
        targetRevision: HEAD
        path: 'addons/{{metadata.labels.region}}'
      destination:
        server: '{{server}}'
        namespace: kube-system

This single ApplicationSet deploys the appropriate addons to every cluster labeled environment: production, using each cluster’s region label to select the correct path in the repository.

Sync Strategies and Waves

When deploying complex applications with dependencies between resources, you need to control the order of deployment. ArgoCD provides two mechanisms:

Sync Phases

Resources are deployed in three phases: PreSync, Sync, and PostSync. Use Sync Hooks for resources that must complete before the main sync proceeds (database migrations, certificate issuance, etc.):

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: your-app:v1.2.3
        command: ["./migrate.sh"]
      restartPolicy: Never

Sync Waves

Within the Sync phase, waves control ordering. Resources with a lower wave number are applied and must become healthy before resources with higher wave numbers are applied:

# Applied first
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

# Applied after wave 1 is healthy
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"

Notifications and Alerting

ArgoCD Notifications is a standalone controller that sends alerts when Application state changes. It supports Slack, PagerDuty, GitHub commit status, email, and a dozen other providers. Configure it via the argocd-notifications-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token
  template.app-sync-failed: |
    slack:
      attachments: |
        [{
          "title": "{{.app.metadata.name}}",
          "color": "#E96D76",
          "fields": [{
            "title": "Sync Status",
            "value": "{{.app.status.sync.status}}",
            "short": true
          },{
            "title": "Message",
            "value": "{{range .app.status.conditions}}{{.message}}{{end}}",
            "short": false
          }]
        }]
  trigger.on-sync-failed: |
    - when: app.status.sync.status == 'Unknown'
      send: [app-sync-failed]
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed]

Secret Management with ArgoCD

ArgoCD intentionally has no secret management built in — storing secrets in Git as plain text is never acceptable. The common patterns are:

  • Sealed Secrets (Bitnami) — Encrypts secrets with a cluster-specific key. The encrypted secret can be committed to Git; only the cluster can decrypt it.
  • External Secrets Operator — Syncs secrets from Vault, AWS Secrets Manager, GCP Secret Manager, etc. into Kubernetes secrets. The ArgoCD Application manages the ExternalSecret CRD, not the actual secret value.
  • argocd-vault-plugin — A plugin that replaces placeholder values in manifests with secrets retrieved from Vault at sync time.

The External Secrets Operator approach is the most flexible for teams already using a centralized secrets backend. The Application in ArgoCD deploys ExternalSecret objects, which the ESO controller resolves at runtime without ever touching Git.

Production Best Practices

  • Run ArgoCD in HA mode. Use manifests/ha/install.yaml with 3 replicas of the API server and multiple application controller shards for large clusters (100+ applications).
  • Pin image versions. Never use latest for the ArgoCD image itself. Pin to a specific version and upgrade deliberately.
  • Use the App of Apps pattern for bootstrapping. A single root Application deploys all other Applications. This makes cluster bootstrapping idempotent and reproducible.
  • Separate ArgoCD config from application config. Store ArgoCD Application manifests in a dedicated gitops repository, separate from application source code.
  • Enable resource tracking via annotations. Use application.resourceTrackingMethod: annotation in argocd-cm instead of the default label-based tracking, which can conflict with Helm’s own labels.
  • Set resource limits on ArgoCD controllers. Application controller CPU and memory scale with the number of resources tracked. Monitor and tune accordingly.
  • Restrict auto-sync in production. Consider requiring manual sync approval for production environments even when using GitOps — or at minimum require a PR approval gate before changes reach the target branch.

ArgoCD vs Flux

Flux v2 is the other major GitOps operator. Both are CNCF projects. The main differences in practice:

FeatureArgoCDFlux v2
UIBuilt-in web UINo official UI (use Weave GitOps)
Multi-clusterSingle control plane manages many clustersAgent per cluster, pull model
ApplicationSetsNativeKustomization + HelmRelease
Secret managementPlugin-basedSOPS native integration
Learning curveSteeper (more concepts)Lower (Kubernetes-native CRDs)
CNCF statusGraduatedGraduated

ArgoCD wins when you need the UI, multi-cluster management from a central plane, or have a large operations team that benefits from the visual application topology view. Flux wins when you want a simpler, purely Kubernetes-native approach with better SOPS integration for secret management.

FAQ

Can ArgoCD deploy to the cluster it runs in?

Yes. The https://kubernetes.default.svc destination refers to the local cluster. ArgoCD can manage both its own cluster and external clusters simultaneously.

Does ArgoCD support private Git repositories?

Yes. Configure repository credentials via argocd repo add with SSH keys, HTTPS username/password, or GitHub App credentials. Credentials are stored as Kubernetes secrets in the ArgoCD namespace.

How does ArgoCD handle CRD installation?

CRDs can be managed by ArgoCD, but there is a chicken-and-egg problem: if a CRD is not yet installed, ArgoCD cannot validate resources that use it. The recommended pattern is to put CRDs in wave 1 and dependent resources in wave 2, or to use a separate Application for CRDs.

What is the difference between an Application and an AppProject?

An Application is the unit of deployment — it maps a Git source to a cluster destination. An AppProject is a grouping and access control boundary — it restricts what sources and destinations an Application within the project can use. Every Application belongs to exactly one AppProject.

How do I roll back a deployment with ArgoCD?

The GitOps way: revert the commit in Git and let ArgoCD reconcile. ArgoCD also provides a UI-based rollback to any previous sync revision, but this is considered a temporary measure — the Git history should always be updated to match.

Getting Started

The fastest path from zero to a working ArgoCD setup on a local cluster:

# 1. Create a local cluster (kind or minikube)
kind create cluster --name argocd-demo

# 2. Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# 3. Wait for pods
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=120s

# 4. Get the initial admin password
argocd admin initial-password -n argocd

# 5. Port-forward and log in
kubectl port-forward svc/argocd-server -n argocd 8080:443 &
argocd login localhost:8080 --username admin --insecure

# 6. Deploy your first application
argocd app create guestbook 
  --repo https://github.com/argoproj/argocd-example-apps.git 
  --path guestbook 
  --dest-server https://kubernetes.default.svc 
  --dest-namespace guestbook 
  --sync-policy automated

From here, the natural next steps are integrating ArgoCD with your existing CI pipeline (CI builds and pushes the image, updates the image tag in Git, ArgoCD detects the change and syncs), configuring SSO via Dex, and setting up the App of Apps pattern for managing multiple applications declaratively.

For teams looking to go deeper on GitOps and ArgoCD in production, the Kubernetes architecture patterns guide covers how ArgoCD fits into a broader platform engineering stack alongside service mesh, policy enforcement, and observability tooling.

Istio ServiceEntry Explained: External Services, DNS, and Traffic Control

Istio ServiceEntry Explained: External Services, DNS, and Traffic Control

Every production Kubernetes cluster talks to the outside world. Your services call payment APIs, connect to managed databases, push events to SaaS analytics platforms, and reach legacy systems that will never run inside the mesh. By default, Istio lets all outbound traffic flow freely — or blocks it entirely if you flip outboundTrafficPolicy to REGISTRY_ONLY. Neither extreme gives you what you actually need: selective, observable, policy-controlled access to external services.

That is exactly what Istio ServiceEntry solves. It registers external endpoints in the mesh’s internal service registry so that Envoy sidecars can apply the same traffic management, security, and observability features to outbound calls that you already enjoy for east-west traffic. No new proxies, no egress gateways required for the basic case — just a YAML resource that tells the mesh “this external thing exists, and here is how to reach it.”

In this guide, I will walk through every field of the ServiceEntry spec, explain the four DNS resolution modes with real-world use cases, and show production-ready patterns for external APIs, databases, TCP services, and legacy workloads. We will also cover how to combine ServiceEntry with DestinationRule and VirtualService to get circuit breaking, retries, connection pooling, and even sticky sessions for external dependencies.

What Is a ServiceEntry

Istio maintains an internal service registry that merges Kubernetes Services with any additional entries you declare. When a sidecar proxy needs to decide how to route a request, it consults this registry. Services inside the mesh are automatically registered. Services outside the mesh are not — unless you create a ServiceEntry.

A ServiceEntry is a custom resource that adds an entry to the mesh’s service registry. Once registered, the external service becomes a first-class citizen: Envoy generates clusters, routes, and listeners for it, which means you get metrics (istio_requests_total), access logs, distributed traces, mTLS origination, retries, timeouts, circuit breaking — the full Istio feature set.

Without a ServiceEntry, outbound traffic to an external host either passes through as a raw TCP connection (in ALLOW_ANY mode) with no telemetry, or gets dropped with a 502/503 (in REGISTRY_ONLY mode). Both outcomes are undesirable in production. The ServiceEntry bridges that gap.

ServiceEntry Anatomy: All Fields Explained

Let us look at a complete ServiceEntry and then break down each field.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-api
  namespace: production
spec:
  hosts:
    - api.stripe.com
  location: MESH_EXTERNAL
  ports:
    - number: 443
      name: https
      protocol: TLS
  resolution: DNS
  exportTo:
    - "."
    - "istio-system"

hosts

A list of hostnames associated with the service. For external services, this is typically the DNS name your application uses (e.g., api.stripe.com). For services using HTTP protocols, the hosts field is matched against the HTTP Host header. For non-HTTP protocols and services without a DNS name, you can use a synthetic hostname and pair it with addresses or static endpoints.

addresses

Optional virtual IP addresses associated with the service. Useful for TCP services where you want to assign a VIP that the sidecar will intercept. Not required for HTTP/HTTPS services that use hostname-based routing.

ports

The ports on which the external service is exposed. Each port needs a number, name, and protocol. The protocol matters: setting it to TLS tells Envoy to perform SNI-based routing without terminating TLS. Setting it to HTTPS means HTTP over TLS. For databases, you’ll typically use TCP.

location

MESH_EXTERNAL or MESH_INTERNAL. Use MESH_EXTERNAL for services outside your cluster (third-party APIs, managed databases). Use MESH_INTERNAL for services inside your infrastructure that are not part of the mesh — for example, VMs running in the same VPC that do not have a sidecar, or a Kubernetes Service in a namespace without injection enabled. The location affects how mTLS is applied and how metrics are labeled.

resolution

How the sidecar resolves the endpoint addresses. This is the most critical field and I will dedicate the next section to it. Options: NONE, STATIC, DNS, DNS_ROUND_ROBIN.

endpoints

An explicit list of network endpoints. Required when resolution is STATIC. Optional with DNS resolution to provide labels or locality information. Each endpoint can have an address, ports, labels, network, locality, and weight.

exportTo

Controls the visibility of this ServiceEntry across namespaces. Use "." for the current namespace only, "*" for all namespaces. In multi-team clusters, restrict exports to avoid namespace pollution.

Resolution Types: NONE vs STATIC vs DNS vs DNS_ROUND_ROBIN

The resolution field determines how Envoy discovers the IP addresses behind the service. Getting this wrong is the number one cause of ServiceEntry misconfigurations. Here is a clear breakdown.

ResolutionHow It WorksBest For
NONEEnvoy uses the original destination IP from the connection. No DNS lookup by the proxy.Wildcard entries, pass-through scenarios, services where the application already resolved the IP.
STATICEnvoy routes to the IPs listed in the endpoints field. No DNS involved.Services with stable, known IPs (e.g., on-prem databases, VMs with fixed IPs).
DNSEnvoy resolves the hostname at connection time and creates an endpoint per returned IP. Uses async DNS with health checking per IP.External APIs behind load balancers, managed databases with DNS endpoints (RDS, CloudSQL).
DNS_ROUND_ROBINEnvoy resolves the hostname and uses a single logical endpoint, rotating across returned IPs. No per-IP health checking.Simple external services, services where you do not need per-endpoint circuit breaking.

When to Use NONE

Use NONE when you want to register a range of external IPs or wildcard hosts without Envoy performing any address resolution. This is common for broad egress policies: “allow traffic to *.googleapis.com on port 443.” Envoy will simply forward traffic to whatever IP the application resolved via kube-dns. The downside: Envoy has limited ability to apply per-endpoint policies.

When to Use STATIC

Use STATIC when the external service has known, stable IP addresses that rarely change. This avoids DNS dependencies entirely. You define the IPs in the endpoints list. Classic use case: a legacy Oracle database on a fixed IP in your data center.

When to Use DNS

Use DNS for most external API integrations. Envoy performs asynchronous DNS resolution and creates a cluster endpoint for each returned IP address. This enables per-endpoint health checking and circuit breaking — critical for production reliability. This is the mode you want for services like api.stripe.com or your RDS instance endpoint.

When to Use DNS_ROUND_ROBIN

Use DNS_ROUND_ROBIN when the external hostname returns many IPs and you do not need per-IP circuit breaking. Envoy treats all resolved IPs as a single logical endpoint and round-robins across them. This is lighter weight than DNS mode and avoids creating a large number of endpoints in Envoy’s cluster configuration.

Practical Patterns

Pattern 1: External HTTP API (api.stripe.com)

The most common ServiceEntry pattern. Your application calls a third-party HTTPS API. You want Istio telemetry, and optionally retries and timeouts.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: stripe-api
  namespace: payments
spec:
  hosts:
    - api.stripe.com
  location: MESH_EXTERNAL
  ports:
    - number: 443
      name: tls
      protocol: TLS
  resolution: DNS

Note the protocol is TLS, not HTTPS. Since your application initiates the TLS handshake directly, Envoy handles this as opaque TLS using SNI-based routing. If you were terminating TLS at the sidecar and doing TLS origination via a DestinationRule, you would set the protocol to HTTP and handle the upgrade separately — but for most external APIs, let the application manage its own TLS.

Pattern 2: External Managed Database (RDS / CloudSQL)

Managed databases expose a DNS endpoint that resolves to one or more IPs. During failover, the DNS record changes. You need Envoy to respect DNS TTLs and route to the current primary.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: orders-database
  namespace: orders
spec:
  hosts:
    - orders-db.abc123.us-east-1.rds.amazonaws.com
  location: MESH_EXTERNAL
  ports:
    - number: 5432
      name: postgres
      protocol: TCP
  resolution: DNS

For TCP services, Envoy cannot use HTTP headers to route, so it relies on IP-based matching. The DNS resolution mode ensures Envoy periodically re-resolves the hostname and updates its endpoint list. This is critical for RDS multi-AZ failover scenarios where the DNS endpoint flips to a new IP.

Pattern 3: Legacy Internal Service Not in the Mesh

You have a monitoring service running on a set of VMs at known IP addresses inside your VPC. It is not part of the mesh, but your meshed services need to talk to it.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: legacy-monitoring
  namespace: observability
spec:
  hosts:
    - legacy-monitoring.internal
  location: MESH_INTERNAL
  ports:
    - number: 8080
      name: http
      protocol: HTTP
  resolution: STATIC
  endpoints:
    - address: 10.0.5.10
    - address: 10.0.5.11
    - address: 10.0.5.12

Key differences: location is MESH_INTERNAL because the service lives inside your network, and resolution is STATIC because we know the IPs. The hostname legacy-monitoring.internal is synthetic — your application uses it, and Istio’s DNS proxy (or a CoreDNS entry) resolves it to one of the listed endpoints.

Pattern 4: TCP Services with Multiple Ports

Some external services expose multiple TCP ports — for example, an Elasticsearch cluster with both data (9200) and transport (9300) ports.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-elasticsearch
  namespace: search
spec:
  hosts:
    - es.example.com
  location: MESH_EXTERNAL
  ports:
    - number: 9200
      name: http
      protocol: HTTP
    - number: 9300
      name: transport
      protocol: TCP
  resolution: DNS

Each port gets its own Envoy listener configuration. The HTTP port benefits from full Layer 7 telemetry and traffic management. The TCP port gets Layer 4 metrics and connection-level policies.

Combining ServiceEntry with DestinationRule

A ServiceEntry alone registers the external service. To apply traffic policies — connection pooling, circuit breaking, TLS origination, load balancing — you pair it with a DestinationRule. This is where things get powerful.

Connection Pooling and Circuit Breaking

External APIs have rate limits. Your managed database has a maximum connection count. Protecting these dependencies at the mesh level prevents cascading failures.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: stripe-api
  namespace: payments
spec:
  hosts:
    - api.stripe.com
  location: MESH_EXTERNAL
  ports:
    - number: 443
      name: tls
      protocol: TLS
  resolution: DNS
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: stripe-api-dr
  namespace: payments
spec:
  host: api.stripe.com
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 50
        connectTimeout: 5s
      http:
        h2UpgradePolicy: DO_NOT_UPGRADE
        maxRequestsPerConnection: 100
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 30s
      baseEjectionTime: 60s
      maxEjectionPercent: 100

This configuration caps outbound connections to Stripe at 50, sets a 5-second connection timeout, and ejects endpoints that return 3 consecutive 5xx errors. In production, this prevents a degraded third-party API from consuming all your connection slots and causing a domino effect across your services.

TLS Origination

Sometimes your application speaks plain HTTP, but the external service requires HTTPS. Instead of modifying application code, you can offload TLS origination to the sidecar.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-api
  namespace: default
spec:
  hosts:
    - api.external-service.com
  location: MESH_EXTERNAL
  ports:
    - number: 80
      name: http
      protocol: HTTP
    - number: 443
      name: https
      protocol: TLS
  resolution: DNS
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: external-api-tls
  namespace: default
spec:
  host: api.external-service.com
  trafficPolicy:
    portLevelSettings:
      - port:
          number: 443
        tls:
          mode: SIMPLE

Your application sends HTTP to port 80. A VirtualService (shown in the next section) redirects that to port 443. The DestinationRule initiates TLS to the external endpoint. The application never knows TLS happened.

Combining ServiceEntry with VirtualService

VirtualService gives you Layer 7 traffic management for external services: retries, timeouts, fault injection, header-based routing, and traffic shifting. This is invaluable when you are migrating between API providers or need resilience policies for unreliable external dependencies.

Retries and Timeouts

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: stripe-api-vs
  namespace: payments
spec:
  hosts:
    - api.stripe.com
  http:
    - route:
        - destination:
            host: api.stripe.com
            port:
              number: 443
      timeout: 10s
      retries:
        attempts: 3
        perTryTimeout: 3s
        retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
        retryRemoteLocalities: true

This applies a 10-second overall timeout with up to 3 retry attempts (3 seconds each) for specific failure conditions. Note that this only works for HTTP-protocol ServiceEntries. For TLS-protocol entries where Envoy cannot see the HTTP layer, you are limited to TCP-level connection retries configured via the DestinationRule.

Traffic Shifting Between External Providers

Migrating from one external API to another? Use weighted routing to shift traffic gradually.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: geocoding-primary
  namespace: geo
spec:
  hosts:
    - geocoding.internal
  location: MESH_EXTERNAL
  ports:
    - number: 443
      name: tls
      protocol: TLS
  resolution: STATIC
  endpoints:
    - address: api.old-geocoding-provider.com
      labels:
        provider: old
    - address: api.new-geocoding-provider.com
      labels:
        provider: new
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: geocoding-dr
  namespace: geo
spec:
  host: geocoding.internal
  trafficPolicy:
    tls:
      mode: SIMPLE
  subsets:
    - name: old-provider
      labels:
        provider: old
    - name: new-provider
      labels:
        provider: new
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: geocoding-vs
  namespace: geo
spec:
  hosts:
    - geocoding.internal
  http:
    - route:
        - destination:
            host: geocoding.internal
            subset: old-provider
          weight: 80
        - destination:
            host: geocoding.internal
            subset: new-provider
          weight: 20

This sends 80% of geocoding traffic to the old provider and 20% to the new one. Adjust the weights as you gain confidence. Fully reversible — just set the old provider back to 100%.

DNS Resolution Patterns: Istio DNS Proxy vs kube-dns

Istio DNS resolution for external services involves two layers: how your application resolves the hostname (kube-dns / CoreDNS), and how the sidecar resolves the hostname (Envoy’s async DNS or Istio’s DNS proxy). Understanding the interplay is crucial for reliable Istio DNS behavior.

Default Flow (Without Istio DNS Proxy)

Your application calls api.stripe.com. kube-dns resolves it to an IP. The application opens a connection to that IP. The sidecar intercepts the connection and — if the ServiceEntry uses DNS resolution — Envoy independently resolves api.stripe.com to determine its endpoint list. Two separate DNS lookups happen, which can lead to inconsistencies if DNS records change between the two resolutions.

With Istio DNS Proxy (dns.istio.io)

Istio’s sidecar includes a DNS proxy that intercepts DNS queries from the application. When enabled (via meshConfig.defaultConfig.proxyMetadata.ISTIO_META_DNS_CAPTURE and ISTIO_META_DNS_AUTO_ALLOCATE), the proxy can:

  • Auto-allocate virtual IPs for ServiceEntry hosts that do not have addresses defined, which is critical for TCP ServiceEntries that need IP-based matching.
  • Resolve ServiceEntry hosts directly, avoiding the round-trip to kube-dns for known mesh services.
  • Ensure consistency between the application’s DNS resolution and the sidecar’s endpoint resolution.

In modern Istio installations (1.18+), DNS capture is enabled by default. Verify with:

istioctl proxy-config bootstrap <pod-name> -n <namespace> | grep -A2 "ISTIO_META_DNS"

When DNS Proxy Matters Most

The DNS proxy is especially important for TCP ServiceEntries without an explicit addresses field. Without a VIP, Envoy cannot match an incoming TCP connection to the correct ServiceEntry because there is no HTTP Host header to inspect. The DNS proxy solves this by auto-allocating a VIP from the 240.240.0.0/16 range and returning that VIP when the application resolves the hostname. The sidecar then intercepts traffic to that VIP and routes it to the correct external endpoint.

Sticky Sessions with ServiceEntry

Some external services require session affinity — for example, a legacy service that stores session state in memory, or a WebSocket endpoint that must maintain a persistent connection to the same backend. Istio supports sticky sessions for external services through consistent hashing in a DestinationRule.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: legacy-session-service
  namespace: default
spec:
  hosts:
    - legacy-session.internal
  location: MESH_INTERNAL
  ports:
    - number: 8080
      name: http
      protocol: HTTP
  resolution: STATIC
  endpoints:
    - address: 10.0.1.10
    - address: 10.0.1.11
    - address: 10.0.1.12
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: legacy-session-dr
  namespace: default
spec:
  host: legacy-session.internal
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: SERVERID
          ttl: 3600s

This configuration hashes on an HTTP cookie named SERVERID. If the cookie does not exist, Envoy generates one and sets it on the response so that subsequent requests from the same client stick to the same endpoint. You can also hash on:

  • HTTP header: consistentHash.httpHeaderName: "x-user-id" — useful when your application sends a user identifier in every request.
  • Source IP: consistentHash.useSourceIp: true — simplest option but breaks in environments with NAT or shared egress IPs.
  • Query parameter: consistentHash.httpQueryParameterName: "session_id" — for REST APIs that include a session identifier in the URL.

Sticky sessions with ServiceEntry work identically to in-mesh sticky sessions. The key requirement is that the ServiceEntry must use STATIC or DNS resolution (not NONE) so that Envoy has multiple endpoints to hash across. With DNS_ROUND_ROBIN, there is only one logical endpoint, so consistent hashing has no effect.

Troubleshooting Common Issues

503 Errors When Calling External Services

The most common ServiceEntry issue. Start with this diagnostic sequence:

# Check if the ServiceEntry is applied and visible to the proxy
istioctl proxy-config cluster <pod-name> -n <namespace> | grep <external-host>

# Check the listeners
istioctl proxy-config listener <pod-name> -n <namespace> --port <port>

# Look at Envoy access logs for the specific request
kubectl logs <pod-name> -n <namespace> -c istio-proxy | grep <external-host>

Common causes of 503 errors:

  • Wrong protocol: Setting protocol: HTTPS when your application initiates TLS. Use TLS for pass-through; use HTTP only if the sidecar does TLS origination.
  • Missing ServiceEntry in REGISTRY_ONLY mode: If outboundTrafficPolicy is REGISTRY_ONLY, any host without a ServiceEntry is blocked.
  • exportTo restriction: The ServiceEntry is in namespace A, exported only to ".", and the calling pod is in namespace B.
  • DNS resolution failure: Envoy cannot resolve the hostname. Check that the DNS servers are reachable from the pod.

DNS Resolution Failures

When Envoy’s async DNS resolver fails, you will see UH (upstream unhealthy) or UF (upstream connection failure) flags in access logs.

# Verify DNS works from inside the sidecar
kubectl exec <pod-name> -n <namespace> -c istio-proxy -- \
  pilot-agent request GET /dns_resolve?proxyID=<pod-name>.<namespace>&host=api.stripe.com

# Check Envoy cluster health
istioctl proxy-config endpoint <pod-name> -n <namespace> | grep <external-host>

If the endpoint shows UNHEALTHY, Envoy resolved the DNS but the outlier detection ejected the host. If no endpoint appears at all, DNS resolution is failing. Common fix: ensure your pods can reach an external DNS server, or that CoreDNS is configured to forward queries for the external domain.

TLS Origination Not Working

If you configured TLS origination via a DestinationRule but traffic still fails:

  • Ensure the ServiceEntry port protocol is HTTP, not TLS. If you set it to TLS, Envoy treats the connection as opaque TLS pass-through and will not apply the DestinationRule’s TLS settings.
  • Verify the DestinationRule’s host field exactly matches the ServiceEntry’s hosts entry.
  • Check that the VirtualService (if used) routes to the correct port number.

TCP ServiceEntry Not Intercepting Traffic

For TCP-protocol ServiceEntries without the DNS proxy, Envoy cannot match traffic by hostname. You must either:

  • Set an explicit addresses field with a VIP that your application targets.
  • Enable Istio’s DNS proxy to auto-allocate VIPs.
  • Ensure the destination IP matches what the ServiceEntry resolves to.

Without one of these, TCP traffic goes through the PassthroughCluster and bypasses your ServiceEntry entirely.

Frequently Asked Questions

Do I need a ServiceEntry if outboundTrafficPolicy is set to ALLOW_ANY?

You do not need one for connectivity — your services can reach external hosts without it. But you should create ServiceEntries anyway. Without them, outbound traffic goes through the PassthroughCluster, which means no detailed metrics per destination, no access logging with the external hostname, no circuit breaking, no retries, and no timeout policies. A ServiceEntry is the difference between “it works” and “it works reliably with observability.”

What is the difference between protocol TLS and HTTPS in a ServiceEntry port?

TLS tells Envoy to treat the connection as opaque TLS. Envoy reads the SNI header to determine routing but does not decrypt the payload. Use this when your application initiates TLS directly. HTTPS tells Envoy the protocol is HTTP over TLS, which implies Envoy should handle TLS. In practice, for external services where the application manages its own TLS, use TLS. Use HTTP with a DestinationRule TLS origination when you want the sidecar to handle TLS.

Can I use wildcards in ServiceEntry hosts?

Yes, but with limitations. You can use *.example.com to match any subdomain of example.com. However, wildcard entries only work with resolution: NONE because Envoy cannot perform DNS lookups for wildcard hostnames. This means you lose the ability to apply per-endpoint traffic policies. Wildcard ServiceEntries are best used for broad egress access control rather than fine-grained traffic management.

How do I configure sticky sessions for an external service behind a ServiceEntry?

Create a ServiceEntry with STATIC or DNS resolution (so Envoy has multiple endpoints), then pair it with a DestinationRule that configures consistentHash under trafficPolicy.loadBalancer. You can hash on an HTTP cookie, header, source IP, or query parameter. The ServiceEntry must expose multiple endpoints for consistent hashing to have any effect. See the “Sticky Sessions with ServiceEntry” section above for a complete YAML example.

How does ServiceEntry interact with NetworkPolicy and Istio AuthorizationPolicy?

A ServiceEntry does not bypass Kubernetes NetworkPolicy. If a NetworkPolicy blocks egress to the external IP, traffic will be dropped at the CNI level before Envoy can route it. Istio AuthorizationPolicy can also restrict which workloads are allowed to call specific ServiceEntry hosts. For defense in depth, use ServiceEntry for traffic management and observability, AuthorizationPolicy for workload-level access control, and NetworkPolicy for network-level enforcement.

Wrapping Up

ServiceEntry is one of the most practical Istio resources you will use in production. It transforms opaque outbound connections into managed, observable, policy-controlled traffic — and it does so without requiring changes to your application code. Start with the basics: create a ServiceEntry for each external dependency, set the correct resolution type, and pair it with a DestinationRule for connection limits and circuit breaking. As you mature, add VirtualServices for retries and timeouts, configure sticky sessions where needed, and enable the DNS proxy for seamless TCP service integration.

The pattern is always the same: register the service, apply policies, observe the traffic. Every external dependency you formalize with a ServiceEntry is one fewer blind spot in your production mesh.

Prometheus Alertmanager vs Grafana Alerting (2026): Architecture, Features, and When to Use Each

Prometheus Alertmanager vs Grafana Alerting (2026): Architecture, Features, and When to Use Each

Most observability stacks that have been running in production for more than a year end up with alerting spread across two systems: Prometheus Alertmanager handling metric-based alerts and Grafana Alerting managing everything else. Engineers add a Slack integration in Grafana because it is convenient, then realize their Alertmanager routing tree already covers the same service. Before long, the on-call team receives duplicated pages, silencing rules live in two places, and nobody is confident which system is authoritative.

Related reading: OpenSearch vs Loki vs Quickwit vs ClickHouse.

This is the alerting consolidation problem, and it affects teams of every size. The question is straightforward: should you standardize on Prometheus Alertmanager, move everything into Grafana Alerting, or deliberately run both? The answer depends on your datasource mix, your GitOps maturity, and how your organization manages on-call routing. This guide breaks down the architecture, features, and operational trade-offs of each system so you can make a deliberate choice instead of drifting into accidental complexity.

Alertmanager vs Grafana Alerting: the short answer

Prometheus Alertmanager only routes and delivers alerts that Prometheus has already evaluated — it is a single YAML file, GitOps-native, lightweight, and unbeatable for a Prometheus-only metrics stack. Grafana Alerting evaluates the rules itself against any datasource (Loki, Elasticsearch, CloudWatch, SQL, Prometheus), adds a UI, native multi-tenancy and mute timings, and is the only choice when you alert on more than metrics. If your alerts are all PromQL and you value declarative config, use Alertmanager; if you alert on logs, databases or want UI-driven management, use Grafana Alerting; many teams deliberately run both — Alertmanager for the metric pipeline, Grafana for everything else. That is the same question as “Grafana alerts vs Prometheus alerts”, and the rest of this guide backs up each side with architecture, a feature table, and a decision framework.

Architecture Overview

Before comparing features, you need to understand how each system fits into the alerting pipeline. They occupy the same logical space — “receive a condition, route a notification” — but they get there from fundamentally different starting points.

Prometheus Alertmanager: The Standalone Receiver

Alertmanager is a dedicated, standalone component in the Prometheus ecosystem. It does not evaluate alert rules itself. Instead, Prometheus (or any compatible sender like Thanos Ruler, Cortex, or Mimir Ruler) evaluates PromQL expressions and pushes firing alerts to the Alertmanager API. Alertmanager then handles deduplication, grouping, inhibition, silencing, and notification delivery. Those metrics reach Prometheus in the first place through the scrape targets you declare as ServiceMonitor or PodMonitor CRDs.

# Simplified Prometheus → Alertmanager flow
#
# [Prometheus] --evaluates rules--> [firing alerts]
#        |
#        +--POST /api/v2/alerts--> [Alertmanager]
#                                      |
#                          +-----------+-----------+
#                          |           |           |
#                       [Slack]    [PagerDuty]  [Email]

The entire configuration lives in a single YAML file (alertmanager.yml). This includes the routing tree, receiver definitions, inhibition rules, and silence templates. There is no database, no UI-driven state — just a config file and an optional local storage directory for notification state and silences. This makes it trivially reproducible and ideal for GitOps workflows.

For high availability, you run multiple Alertmanager instances in a gossip-based cluster. They use a mesh protocol to share silence and notification state, ensuring that failover does not result in duplicate or lost notifications. The HA model is well-understood and has been stable for years.

Grafana Alerting: The Integrated Platform

Grafana Alerting (sometimes called “Grafana Unified Alerting,” introduced in Grafana 8 and significantly matured through Grafana 11 and 12) takes a different architectural approach. It embeds the entire alerting lifecycle — rule evaluation, state management, routing, and notification — inside the Grafana server process. Under the hood, it actually uses a fork of Alertmanager for the routing and notification layer, but this is an implementation detail that is invisible to users.

# Simplified Grafana Alerting flow
#
# [Grafana Server]
#   ├── Rule Evaluation Engine
#   │     ├── queries Prometheus
#   │     ├── queries Loki
#   │     ├── queries CloudWatch
#   │     └── queries any supported datasource
#   │
#   ├── Alert State Manager (internal)
#   │
#   └── Embedded Alertmanager (routing + notifications)
#           |
#           +-----------+-----------+
#           |           |           |
#        [Slack]    [PagerDuty]  [Email]

The critical distinction is that Grafana Alerting evaluates alert rules itself, querying any configured datasource — not just Prometheus. It can fire alerts based on Loki log queries, Elasticsearch searches, CloudWatch metrics, PostgreSQL queries, or any of the 100+ datasource plugins available in Grafana. Rule definitions, contact points, notification policies, and mute timings are stored in the Grafana database (or provisioned via YAML files and the Grafana API).

For high availability in self-hosted environments, Grafana Alerting relies on a shared database and a peer-discovery mechanism between Grafana instances. In Grafana Cloud, HA is fully managed by Grafana Labs.

A note on the term “Grafana Alertmanager”

People search for “Grafana Alertmanager” and mean one of three different things, so it is worth separating them. First, the embedded Alertmanager: Grafana Alerting uses a fork of the Prometheus Alertmanager code internally for its routing and notification layer — when the Grafana UI shows an “Alertmanager”, this built-in engine is usually what it means. Second, an external Alertmanager: Grafana can connect to a standalone Prometheus Alertmanager as a datasource and act as a read-only viewer for its firing alerts and silences (see the hybrid section below). Third, Mimir/Cloud Alertmanager: Grafana Cloud exposes a managed, multi-tenant Alertmanager for teams that want Prometheus-native routing without self-hosting. So “Grafana Alertmanager” is not a separate product — it is one of these three, and which one you mean changes the answer to almost every configuration question.

Feature Comparison

The following table provides a side-by-side comparison of the capabilities that matter most in production alerting systems. Both systems are mature, but they prioritize different things.

FeaturePrometheus AlertmanagerGrafana Alerting
DatasourcesPrometheus-compatible only (Prometheus, Thanos, Mimir, VictoriaMetrics)Any Grafana datasource (Prometheus, Loki, Elasticsearch, CloudWatch, SQL databases, etc.)
Rule evaluationExternal (Prometheus/Ruler evaluates rules and pushes alerts)Built-in (Grafana evaluates rules directly)
Routing treeHierarchical YAML-based routing with match/match_re, continue, group_byNotification policies with label matchers, nested policies, mute timings
GroupingFull support via group_by, group_wait, group_intervalFull support via notification policies with equivalent controls
InhibitionNative inhibition rules (suppress alerts when a related alert is firing)Supported since Grafana 10.3 but less flexible than Alertmanager
SilencingLabel-based silences via API or UI, time-limitedMute timings (recurring schedules) and silences (ad-hoc, label-based)
Notification channelsEmail, Slack, PagerDuty, OpsGenie, VictoriaOps, webhook, WeChat, Telegram, SNS, WebexAll of the above plus Teams, Discord, Google Chat, LINE, Threema, Oncall, and more via contact points
TemplatingGo templates in notification configGo templates with access to Grafana template variables and functions
Multi-tenancyNot built-in; achieved via separate instances or Mimir AlertmanagerNative multi-tenancy via Grafana organizations and RBAC
High availabilityGossip-based cluster (peer mesh, well-proven)Database-backed HA with peer discovery between Grafana instances
Configuration modelSingle YAML file, fully declarativeUI + API + provisioning YAML files, stored in database
GitOps compatibilityExcellent — config file lives in version control nativelyPossible via provisioning files or Terraform provider, but requires extra tooling
External alert sourcesAny system that can POST to the Alertmanager APISupported via the Grafana Alerting API (external alerts can be pushed)
Managed serviceAvailable via Grafana Cloud (as Mimir Alertmanager), Amazon Managed PrometheusAvailable via Grafana Cloud

Alertmanager Strengths

Alertmanager has been a production staple since 2015. Over a decade of use across thousands of organizations has made it one of the most battle-tested components in the CNCF ecosystem. Here is where it genuinely excels.

Declarative, GitOps-Native Configuration

The entire Alertmanager configuration is a single YAML file. There is no hidden state in a database, no click-driven configuration that someone forgets to document. You check it into Git, review it in a pull request, and deploy it through your CI/CD pipeline like any other infrastructure code. This is a significant operational advantage for teams that have invested in GitOps.

# alertmanager.yml — everything in one file
global:
  resolve_timeout: 5m
  slack_api_url: "https://hooks.slack.com/services/T00/B00/XXX"

route:
  receiver: platform-team
  group_by: [alertname, cluster, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: pagerduty-oncall
      group_wait: 10s
    - match_re:
        team: "^(payments|checkout)$"
      receiver: payments-slack
      continue: true

receivers:
  - name: platform-team
    slack_configs:
      - channel: "#platform-alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: ""
  - name: payments-slack
    slack_configs:
      - channel: "#payments-oncall"

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: [alertname, cluster]

Every change is auditable. Rollbacks are a git revert away. This matters enormously when you are debugging why an alert did not fire at 3 AM.

Lightweight and Single-Purpose

Alertmanager does one thing: route and deliver notifications. It has no dashboard, no query engine, no datasource plugins. This single-purpose design makes it operationally simple. Resource consumption is minimal — a small Alertmanager instance handles thousands of active alerts on a few hundred megabytes of memory. It starts in milliseconds and requires almost no maintenance.

Mature Inhibition and Routing

Alertmanager’s inhibition rules are first-class citizens. You can suppress downstream warnings when a critical alert is already firing, preventing alert storms from overwhelming your on-call team. The hierarchical routing tree with continue flags allows for nuanced delivery: send to the team channel AND escalate to PagerDuty simultaneously, with different grouping strategies at each level.

Proven High Availability

The gossip-based HA cluster has been stable for years. Running three Alertmanager replicas behind a load balancer (or using Kubernetes service discovery) gives you reliable notification delivery without shared storage. The protocol handles deduplication across instances automatically, which is the hardest part of distributed alerting.

Grafana Alerting Strengths

Grafana Alerting has matured considerably since its rocky introduction in Grafana 8. By Grafana 11 and 12, it has become a legitimate production alerting platform with capabilities that Alertmanager cannot match on its own.

Multi-Datasource Alert Rules

This is Grafana Alerting’s strongest differentiator. You can write alert rules that query Loki for error log spikes, CloudWatch for AWS resource utilization, Elasticsearch for application errors, or a PostgreSQL database for business metrics — all from the same alerting system. If your observability stack includes more than just Prometheus, this eliminates the need for separate alerting tools per datasource.

# Grafana alert rule provisioning example — alerting on Loki log errors
apiVersion: 1
groups:
  - orgId: 1
    name: application-errors
    folder: Production
    interval: 1m
    rules:
      - uid: loki-error-spike
        title: "High error rate in payment service"
        condition: C
        data:
          - refId: A
            datasourceUid: loki-prod
            model:
              expr: 'sum(rate({app="payment-service"} |= "ERROR" [5m]))'
          - refId: B
            datasourceUid: "__expr__"
            model:
              type: reduce
              expression: A
              reducer: last
          - refId: C
            datasourceUid: "__expr__"
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator:
                    type: gt
                    params: [10]
        for: 5m
        labels:
          severity: warning
          team: payments

This is something Alertmanager simply cannot do. Alertmanager only receives pre-evaluated alerts — it has no concept of datasources or query execution.

Unified UI for Alert Management

Grafana provides a single pane of glass for alert rule creation, visualization, notification policy management, contact point configuration, and silence management. For teams where not every engineer is comfortable editing YAML routing trees, the visual notification policy editor significantly reduces the barrier to entry. You can see the state of every alert rule, its evaluation history, and the exact notification path it will take — all without leaving the browser.

Native Multi-Tenancy and RBAC

Grafana’s organization model and role-based access control extend naturally to alerting. Different teams can manage their own alert rules, contact points, and notification policies within their organization or folder scope, without seeing or interfering with other teams. Achieving this with standalone Alertmanager requires either running separate instances per tenant or using Mimir’s multi-tenant Alertmanager.

Mute Timings and Richer Scheduling

While Alertmanager supports silences (ad-hoc, time-limited suppressions), Grafana Alerting adds mute timings — recurring time-based windows where notifications are suppressed. This is useful for scheduled maintenance windows, business-hours-only alerting, or suppressing non-critical alerts on weekends. Alertmanager requires external tooling or manual silence creation for recurring windows.

Grafana Cloud as a Managed Option

For teams that want to avoid managing alerting infrastructure entirely, Grafana Cloud provides a fully managed Grafana Alerting stack. This includes HA, state persistence, and notification delivery without any self-hosted components. The Grafana Cloud alerting stack also includes a managed Mimir Alertmanager, which means you can use Prometheus-native alerting rules if you prefer that model while still benefiting from the managed infrastructure.

When to Use Prometheus Alertmanager

Alertmanager is the right choice when the following conditions describe your environment:

  • Your metrics stack is Prometheus-native. If all your alert rules are PromQL expressions evaluated by Prometheus, Thanos Ruler, or Mimir Ruler, Alertmanager is the natural fit. There is no added value in routing those alerts through Grafana.
  • GitOps is non-negotiable. If every infrastructure change must go through a pull request and be fully declarative, Alertmanager’s single-file configuration model is significantly easier to manage than Grafana’s database-backed state. Tools like amtool provide config validation in CI pipelines.
  • You need fine-grained routing with inhibition. Complex routing trees with multiple levels of grouping, inhibition rules, and continue flags are more naturally expressed in Alertmanager’s YAML format. The routing logic has been stable and well-documented for years.
  • You run microservices with per-team routing. If each team owns its routing subtree and the routing logic is complex, Alertmanager’s hierarchical model scales better than UI-driven configuration. Teams can own their section of the config file via CODEOWNERS in Git.
  • You want minimal operational overhead. Alertmanager is a single binary with minimal resource requirements. There is no database to back up, no migrations to run, and no UI framework to keep updated.

When to Use Grafana Alerting

Grafana Alerting is the right choice when these conditions apply:

  • You alert on more than just Prometheus metrics. If you need alert rules based on Loki logs, Elasticsearch queries, CloudWatch metrics, or database queries, Grafana Alerting is the only option that handles all of these natively. The alternative is running separate alerting tools per datasource, which is worse.
  • Your team prefers UI-driven configuration. Not every engineer wants to edit YAML routing trees. If your organization values a visual interface for managing alerts, contact points, and notification policies, Grafana’s UI is a major productivity advantage.
  • You are using Grafana Cloud. If you are already on Grafana Cloud, using its built-in alerting is the path of least resistance. You get HA, managed notification delivery, and a unified experience without running any additional infrastructure.
  • Multi-tenancy is a requirement. If multiple teams need isolated alerting configurations with RBAC, Grafana’s native organization and folder-based access model is significantly easier to set up than running per-tenant Alertmanager instances.
  • You want mute timings for recurring maintenance windows. If your team regularly needs to suppress alerts during scheduled windows (deploy windows, batch processing hours, weekend non-critical suppression), Grafana’s mute timings feature is more ergonomic than creating and managing recurring silences in Alertmanager.

Running Both Together: The Hybrid Pattern

In practice, many production environments run both Alertmanager and Grafana Alerting. This is not necessarily a mistake — it can be a deliberate architectural choice when done with clear boundaries.

Common Hybrid Architecture

The most common pattern looks like this:

  • Prometheus Alertmanager handles all metric-based alerts. PromQL rules are evaluated by Prometheus or a long-term storage ruler (Thanos, Mimir). Alertmanager owns routing, grouping, and notification for these alerts.
  • Grafana Alerting handles non-Prometheus alerts: log-based alerts from Loki, business metrics from SQL datasources, and cross-datasource correlation rules.

The key to making this work without chaos is establishing clear ownership rules:

# Ownership boundaries for hybrid alerting
#
# Prometheus Alertmanager owns:
#   - All PromQL-based alert rules
#   - Infrastructure alerts (node, kubelet, etcd, CoreDNS)
#   - Application SLO/SLI alerts based on metrics
#
# Grafana Alerting owns:
#   - Log-based alert rules (Loki, Elasticsearch)
#   - Business metric alerts (SQL datasources)
#   - Cross-datasource correlation rules
#   - Alerts for teams that prefer UI-driven management
#
# Shared:
#   - Contact points / receivers use the same Slack channels and PagerDuty services
#   - On-call rotations are managed externally (PagerDuty, Grafana OnCall)

Both systems can deliver to the same notification channels. The critical discipline is ensuring that silencing and maintenance windows are applied in both systems when needed. This is the primary operational cost of the hybrid approach.

Grafana as a Viewer for Alertmanager

Even if you use Alertmanager exclusively for routing and notification, Grafana can serve as a read-only viewer. Grafana natively supports connecting to an external Alertmanager datasource, allowing you to see firing alerts, active silences, and alert groups in the Grafana UI. This gives you the operational visibility of Grafana without moving your alerting logic into it.

# Grafana datasource provisioning for external Alertmanager
apiVersion: 1
datasources:
  - name: Alertmanager
    type: alertmanager
    url: http://alertmanager.monitoring.svc:9093
    access: proxy
    jsonData:
      implementation: prometheus

Migration Considerations

If you are moving from one system to the other, here are the practical considerations to plan for.

Migrating from Alertmanager to Grafana Alerting

  • Rule conversion. Your PromQL-based recording and alerting rules defined in Prometheus rule files need to be recreated as Grafana alert rules. Grafana provides a migration tool that can import Prometheus-format rules, but complex expressions may need manual adjustment.
  • Routing tree translation. Alertmanager’s hierarchical routing tree maps to Grafana’s notification policies, but the semantics are not identical. Test the notification routing thoroughly — the continue flag behavior and default routes may differ.
  • Silence and inhibition migration. Active silences are ephemeral and do not need migration. Inhibition rules need to be recreated in Grafana’s format. Recurring maintenance windows should be converted to mute timings.
  • Run in parallel first. The safest migration strategy is to run both systems in parallel for two to four weeks, sending notifications from both, then cutting over when you have confidence in the Grafana setup. Accept the temporary noise of duplicate alerts — it is far cheaper than missing a critical page during migration.

Migrating from Grafana Alerting to Alertmanager

  • Datasource limitation. You can only migrate alerts that are based on Prometheus-compatible datasources. Alerts querying Loki, Elasticsearch, or SQL datasources have no equivalent in Alertmanager — you will need an alternative solution for those.
  • Rule export. Export Grafana alert rules and convert them to Prometheus-format rule files. The Grafana API (GET /api/v1/provisioning/alert-rules) provides structured output that can be transformed with a script.
  • Contact point mapping. Map Grafana contact points to Alertmanager receivers. The configuration format is different, but the concepts are equivalent.
  • State loss. Alertmanager does not carry over Grafana’s alert evaluation history. You start fresh. Plan for a brief period where alerts may re-fire as Prometheus evaluates rules that were previously managed by Grafana.

Decision Framework

If you want a quick decision path, use this framework:

Start here:
│
├── Do you alert on non-Prometheus datasources (Loki, ES, SQL, CloudWatch)?
│   ├── YES → Grafana Alerting (at least for those datasources)
│   └── NO ↓
│
├── Is GitOps/declarative config a hard requirement?
│   ├── YES → Alertmanager
│   └── NO ↓
│
├── Do you need multi-tenancy with RBAC?
│   ├── YES → Grafana Alerting (or Mimir Alertmanager)
│   └── NO ↓
│
├── Are you on Grafana Cloud?
│   ├── YES → Grafana Alerting (path of least resistance)
│   └── NO ↓
│
└── Default → Alertmanager (simpler, lighter, well-proven)

For many teams, the honest answer is “both” — Alertmanager for the Prometheus-native metric pipeline, Grafana Alerting for everything else. That is a valid architecture as long as the ownership boundaries are documented and the on-call team knows where to look.

Frequently Asked Questions

What is the difference between Alertmanager and Grafana Alerting?

Prometheus Alertmanager is a standalone notification routing engine that receives pre-evaluated alerts from Prometheus and delivers them to receivers like Slack, PagerDuty, or email. It does not evaluate alert rules itself. Grafana Alerting is an integrated alerting platform embedded in Grafana that both evaluates alert rules (querying any supported datasource) and handles notification routing. Alertmanager is configured entirely via YAML, while Grafana Alerting offers a UI, API, and file-based provisioning. The fundamental difference is scope: Alertmanager handles only the routing and notification phase, while Grafana Alerting handles the full lifecycle from query evaluation to notification.

Can Grafana Alerting replace Prometheus Alertmanager?

Yes, for many use cases. Grafana Alerting can evaluate PromQL rules directly against your Prometheus datasource, so you do not strictly need a separate Alertmanager instance. However, there are scenarios where Alertmanager remains the better choice: heavily GitOps-driven environments, teams that need Alertmanager’s mature inhibition rules, or architectures where Prometheus rule evaluation happens externally (Thanos Ruler, Mimir Ruler) and a dedicated Alertmanager is already in the pipeline. If your only datasource is Prometheus and you value declarative configuration, Alertmanager is still simpler and lighter.

Is Grafana Alertmanager the same as Prometheus Alertmanager?

Not exactly. Grafana Alerting uses a fork of the Prometheus Alertmanager code internally for its notification routing engine, but it is not the same product. The Grafana “Alertmanager” you see in the UI is a managed, embedded component with a different configuration interface (notification policies, contact points, mute timings) compared to the standalone Prometheus Alertmanager (routing tree, receivers, inhibition rules in YAML). Grafana can also connect to an external Prometheus Alertmanager as a datasource, which adds to the confusion. When people refer to “Grafana Alertmanager,” they usually mean the embedded routing engine inside Grafana Alerting.

What are the best alternatives to Prometheus Alertmanager?

The most direct alternative is Grafana Alerting, which can receive and route Prometheus alerts while also supporting other datasources. Beyond that, other options include: Grafana OnCall for on-call management and escalation (often used alongside Alertmanager rather than replacing it), PagerDuty or Opsgenie as managed incident response platforms that can receive alerts directly, Keep as an open-source AIOps alert management platform, and Mimir Alertmanager for multi-tenant environments running Grafana Mimir. The choice depends on whether you need an Alertmanager replacement (routing and notification) or a complementary tool for escalation and incident response.

Should I use Prometheus alerts or Grafana alerts for Kubernetes monitoring?

For Kubernetes monitoring specifically, the kube-prometheus-stack (which includes Prometheus, Alertmanager, and a comprehensive set of pre-built alerting rules) remains the industry standard. These rules are PromQL-based and are designed to work with Alertmanager. If you are deploying kube-prometheus-stack, using Alertmanager for metric-based alerts is the straightforward choice. Add Grafana Alerting on top if you also need to alert on logs (via Loki) or non-metric datasources. For Kubernetes-specific monitoring, the combination of Prometheus rules with Alertmanager for routing is the most mature and well-supported path.

Final Thoughts

The Alertmanager vs Grafana Alerting debate is not really about which tool is better — it is about which tool fits your operational context. Alertmanager is simpler, lighter, and more GitOps-friendly. Grafana Alerting is more versatile, more accessible to UI-oriented teams, and the only option if you need multi-datasource alerting. Running both is perfectly valid when the boundaries are clear.

The worst outcome is not picking the “wrong” tool. The worst outcome is running both accidentally, with overlapping coverage, duplicated notifications, and no clear ownership. Whatever you choose, document the decision, define the ownership boundaries, and make sure your on-call team knows exactly where to go when they need to silence an alert at 3 AM.

Gateway API Provider Support in 2026: A Critical Evaluation

Gateway API Provider Support in 2026: A Critical Evaluation

The Kubernetes Gateway API is no longer a future concept—it’s the present standard for traffic management. With the deprecation of Ingress NGINX’s stable APIs signaling a definitive shift, platform teams and architects are now faced with a critical decision: which Gateway API provider to adopt. The official implementations page lists numerous options, but the real-world picture is one of fragmented support, varying stability, and significant gaps that can derail multi-cluster strategies.

In this evaluation, we move beyond marketing checklists to analyze the practical state of Gateway API support across major cloud providers, ingress controllers, and service meshes. We’ll examine which versions are truly production-ready, where the interoperability pitfalls lie, and what you must account for before standardizing across your infrastructure.

The Gateway API Maturity Spectrum: From Experimental to Standard

Not all Gateway API resources are created equal. The API’s unique versioning model—with features progressing through Experimental, Standard, and Extended support tracks—means provider support is inherently uneven. An implementation might fully support the stable Gateway and HTTPRoute resources while offering only partial or experimental backing for GRPCRoute or TCPRoute.

This creates a fundamental challenge for architects: designing for the lowest common denominator or accepting provider-specific constraints. The decision hinges on accurately mapping your traffic management requirements (HTTP, TLS termination, gRPC, TCP/UDP load balancing) against what each provider actually delivers in a stable form.

Core API Support: The Foundation

Most providers now support the v1 (GA) versions of the foundational resources:

  • GatewayClass & Gateway: Nearly universal support for v1. These are the control plane resources for provisioning and configuring load balancers.
  • HTTPRoute: Universal support for v1. This is the workhorse for HTTP/HTTPS traffic routing and is considered the most stable.

However, support for other route types reveals the fragmentation:

  • GRPCRoute: Often in beta or experimental stages. Critical for modern microservices architectures but not yet universally reliable.
  • TCPRoute & UDPRoute: Patchy support. Some providers implement them as beta, others ignore them entirely, forcing fallbacks to provider-specific annotations or custom resources.
  • TLSRoute: Frequently tied to specific certificate management integrations (e.g., cert-manager).

Major Provider Deep Dive: Implementation Realities

AWS Elastic Kubernetes Service (EKS)

AWS offers an official Gateway API controller for EKS. Its support is pragmatic but currently limited:

  • Supported Resources: GatewayClass, Gateway, HTTPRoute, and GRPCRoute (all v1beta1 as of early 2024). Note the use of v1beta1 for GRPCRoute, indicating it’s not yet at GA stability.
  • Underlying Infrastructure: Maps directly to AWS Application Load Balancer (ALB) and Network Load Balancer (NLB). This is a strength (managed AWS services) and a constraint (you inherit ALB/NLB feature limits).
  • Critical Gap: No support for TCPRoute or UDPRoute. If your workload requires raw TCP/UDP load balancing, you must use the legacy Kubernetes Service type LoadBalancer or a different ingress controller alongside the Gateway API controller, creating a disjointed management model.

Google Kubernetes Engine (GKE) & Azure Kubernetes Service (AKS)

Both Google and Azure have integrated Gateway API support directly into their managed Kubernetes offerings, often with a focus on their global load-balancing infrastructures.

  • GKE: Offers the GKE Gateway controller. It supports v1 resources and can provision Google Cloud Global External Load Balancers. Its integration with Google’s certificate management and CDN is a key advantage. However, advanced routing features may require GCP-specific backend configs.
  • AKS: Provides the Application Gateway Ingress Controller (AGIC) with Gateway API support, mapping to Azure Application Gateway. Support for newer route types like GRPCRoute has historically lagged behind other providers.

The pattern here is clear: cloud providers implement the Gateway API as a facade over their existing, proprietary load-balancing products. This ensures stability and performance but can limit portability and advanced cross-provider features.

NGINX & Kong Ingress Controller

These third-party, cluster-based controllers offer a different value proposition: consistency across any Kubernetes distribution, including on-premises.

  • NGINX: With its stable Ingress APIs deprecated in favor of Gateway API, its Gateway API implementation is now the primary path forward. It generally has excellent support for the full range of experimental and standard resources, as it’s not constrained by a cloud vendor’s underlying service. This makes it a strong choice for hybrid or multi-cloud deployments where feature parity is crucial.
  • Kong Ingress Controller: Kong has been an early and comprehensive supporter of the Gateway API, often implementing features quickly. It leverages Kong Gateway’s extensive plugin ecosystem, which can be a major draw but also introduces vendor lock-in.

Critical Gaps for Enterprise Architects

Beyond checking resource support boxes, several deeper gaps can impact production deployments, especially in complex environments.

1. Multi-Cluster & Hybrid Environment Support

The Gateway API specification includes concepts like ReferenceGrant for cross-namespace and future cross-cluster routing. In practice, very few providers have robust, production-ready multi-cluster stories. Most implementations assume a single cluster. If your architecture spans multiple clusters (for isolation, geography, or failure domains), you will likely need to:

  • Manage separate Gateway resources per cluster.
  • Use an external global load balancer (like a cloud DNS/GSLB) to distribute traffic across cluster-specific gateways.
  • This negates some of the API’s promise of a unified, abstracted configuration.

2. Policy Attachment and Extension Consistency

Gateway API is designed to be extended through policy attachment (e.g., for rate limiting, WAF rules, authentication). There is no standard for how these policies are implemented. One provider might use a custom RateLimitPolicy CRD, while another might rely on annotations or a separate policy engine. This creates massive configuration drift and vendor lock-in, breaking the portability goal.

3. Observability and Debugging Interfaces

While the API defines status fields, the richness of operational data—detailed error logs, granular metrics tied to API resources, distributed tracing integration—varies wildly. Some providers expose deep integration with their monitoring stack; others offer minimal visibility. You must verify that the provider’s observability model meets your SRE team’s needs.

Evaluation Framework: Questions for Your Team

Before selecting a provider, work through this technical checklist:

  1. Route Requirements: Do we need stable support for HTTP only, or also gRPC, TCP, UDP? Is beta support acceptable for non-HTTP routes?
  2. Infrastructure Model: Do we want a cloud-managed load balancer (simpler, less control) or a cluster-based controller (more portable, more operational overhead)?
  3. Multi-Cluster Future: Is our architecture single-cluster today but likely to expand? Does the provider have a credible roadmap for multi-cluster Gateway API?
  4. Policy Needs: What advanced policies (auth, WAF, rate limiting) are required? How does the provider implement them? Can we live with vendor-specific policy CRDs?
  5. Observe & Debug: What logging, metrics, and tracing are exposed for Gateway API resources? Do they integrate with our existing observability platform?
  6. Upgrade Path: What is the provider’s track record for supporting new Gateway API releases? How painful are version upgrades?

Strategic Recommendations

Based on the current landscape, here are pragmatic paths forward:

  • For Single-Cloud Deployments: Start with your cloud provider’s native controller (AWS, GKE, AKS). It’s the path of least resistance and best integration with other cloud services (IAM, certificates, monitoring). Just be acutely aware of its specific limitations regarding unsupported route types.
  • For Hybrid/Multi-Cloud or On-Premises: Standardize on a portable, cluster-based controller like Ingress-NGINX or Kong. The consistency across environments will save significant operational complexity, even if it means forgoing some cloud-native integrations.
  • For Greenfield Projects: Design your applications and configurations against the stable v1 resources (Gateway, HTTPRoute) only. Treat any use of beta/experimental resources as a known risk that may require refactoring later.
  • Always Have an Exit Plan: Isolate Gateway API configuration YAMLs from provider-specific policies and annotations. This modularity will make migration less painful when the next generation of providers emerges or when you need to switch.

The Gateway API’s evolution is a net positive for the Kubernetes ecosystem, offering a far more expressive model than the original Ingress. However, in 2026, the provider landscape is still maturing. Support is broad but not deep, and critical gaps in multi-cluster management and policy portability remain. The successful architect will choose a provider not based on a feature checklist, but based on how well its specific constraints and capabilities align with their organization’s immediate traffic patterns and long-term platform strategy. The era of a universal, write-once-run-anywhere Gateway API configuration is not yet here—but with careful, informed provider selection, you can build a robust foundation for it.

Building a Kubernetes Migration Framework: Lessons from Ingress-NGINX

Building a Kubernetes Migration Framework: Lessons from Ingress-NGINX

The recent announcement regarding the deprecation of the Ingress-NGINX controller sent a ripple through the Kubernetes community. For many organizations, it’s the first major deprecation of a foundational, widely-adopted ecosystem component. While the immediate reaction is often tactical—”What do we replace it with?”—the more valuable long-term question is strategic: “How do we systematically manage this and future migrations?”

This event isn’t an anomaly; it’s a precedent. As Kubernetes matures, core add-ons, APIs, and patterns will evolve or sunset. Platform engineering teams need a repeatable, low-risk framework for navigating these changes. Drawing from the Ingress-NGINX transition and established deployment management principles, we can abstract a robust Kubernetes Migration Framework applicable to any major component, from service meshes to CSI drivers.

Why Ad-Hoc Migrations Fail in Production

Attempting a “big bang” replacement or a series of manual, one-off changes is a recipe for extended downtime, configuration drift, and undetected regression. Production Kubernetes environments are complex systems with deep dependencies:

  • Interdependent Workloads: Multiple applications often share the same ingress controller, relying on specific annotations, custom snippets, or behavioral quirks.
  • Automation and GitOps Dependencies: Helm charts, Kustomize overlays, and ArgoCD/Flux manifests are tightly coupled to the existing component’s API and schema.
  • Observability and Security Integration: Monitoring dashboards, logging parsers, and security policies are tuned for the current implementation.
  • Knowledge Silos: Tribal knowledge about workarounds and specific configurations isn’t documented.

A structured framework mitigates these risks by enforcing discipline, creating clear validation gates, and ensuring the capability to roll back at any point.

The Four-Phase Kubernetes Migration Framework

This framework decomposes the migration into four distinct phases: Assessment, Parallel Run, Cutover, and Decommission. Each phase has defined inputs, activities, and exit criteria.

Phase 1: Deep Assessment & Dependency Mapping

Before writing a single line of new configuration, understand the full scope. The goal is to move from “we use Ingress-NGINX” to a precise inventory of how it’s used.

  • Inventory All Ingress Resources: Use kubectl get ingress --all-namespaces as a starting point, but go deeper.
  • Analyze Annotation Usage: Script an analysis to catalog every annotation in use (e.g., nginx.ingress.kubernetes.io/rewrite-target, nginx.ingress.kubernetes.io/configuration-snippet). This reveals functional dependencies.
  • Map to Backend Services: For each Ingress, identify the backend Services and Namespaces. This highlights critical applications and potential blast radius.
  • Review Customizations: Document any custom ConfigMaps for main NGINX configuration, custom template patches, or modifications to the controller deployment itself.
  • Evaluate Alternatives: Based on the inventory, evaluate candidate replacements (e.g., Gateway API with a compatible implementation, another Ingress controller like Emissary-ingress or Traefik). The Google Cloud migration framework provides a useful decision tree for ingress-specific migrations.

The output of this phase is a migration manifesto: a concrete list of what needs to be converted, grouped by complexity and criticality.

Phase 2: Phased Rollout & Parallel Run

This is the core of a low-risk migration. Instead of replacing, you run the new and old systems in parallel, shifting traffic gradually. For ingress, this often means installing the new controller alongside the old one.

  • Dual Installation: Deploy the new ingress controller in the same cluster, configured with a distinct ingress class (e.g., ingressClassName: gateway vs. nginx).
  • Create Canary Ingress Resources: For a low-risk application, create a parallel Ingress or Gateway resource pointing to the new controller. Use techniques like managed deployments with canary patterns to control exposure.
    # Example: A new Gateway API HTTPRoute for a canary service
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
    name: app-canary
    spec:
    parentRefs:
    - name: company-gateway
    rules:
    - backendRefs:
    - name: app-service
    port: 8080
    weight: 10 # Start with 10% of traffic

  • Validate Equivalency: Use traffic mirroring (if supported) or direct synthetic testing against both ingress paths. Compare logs, response headers, latency, and error rates.
  • Iterate and Expand: Gradually increase traffic weight or add more applications to the new stack, group by group, based on the assessment from Phase 1.

This phase relies heavily on your observability stack. Dashboards comparing error rates, latency (p50, p99), and throughput between the old and new paths are essential.

Phase 3: Validation & Automated Cutover

The cutover is not a manual event. It’s the final step in a validation process.

  • Define Validation Tests: Create a suite of tests that must pass before full cutover. This includes:
    • Smoke tests for all critical user journeys.
    • Load tests to verify performance under expected traffic patterns.
    • Security scan validation (e.g., no unintended ports open).
    • Compliance checks (e.g., specific headers are present).
  • Automate the Switch: For each application, the cutover is ultimately a change in its Ingress or Gateway resource. This should be done via your GitOps pipeline. Update the source manifests (e.g., change the ingressClassName), merge, and let automation apply it. This ensures the state is declarative and recorded.
  • Maintain Rollback Capacity: The old system must remain operational and routable (with reduced capacity) during this phase. The GitOps rollback is simply reverting the manifest change.

Phase 4: Observability & Decommission

Once all traffic is successfully migrated and validated over a sustained period (e.g., 72 hours), you can decommission the old component.

  • Monitor Aggressively: Keep a close watch on all key metrics for at least one full business cycle (a week).
  • Remove Old Resources: Delete the old controller’s Deployment, Service, ConfigMaps, and CRDs (if no longer needed).
  • Clean Up Auxiliary Artifacts: Remove old RBAC bindings, service accounts, and any custom monitoring alerts or dashboards specific to the old component.
  • Document Lessons Learned: Update runbooks and architecture diagrams. Note any surprises, gaps in the process, or validation tests that were particularly valuable.

Key Principles for a Resilient Framework

Beyond the phases, these principles should guide your framework’s design:

  • Always Maintain Rollback Capability: Every step should be reversible with minimal disruption. This is a core tenet of managing Kubernetes deployments.
  • Leverage GitOps for State Management: All desired state changes (Ingress resources, controller deployments) must flow through version-controlled manifests. This provides an audit trail, consistency, and the simplest rollback mechanism (git revert).
  • Validate with Production Traffic Patterns: Synthetic tests are insufficient. Use canary weights and traffic mirroring to validate with real user traffic in a controlled manner.
  • Communicate Transparently: Platform teams should maintain a clear migration status page for internal stakeholders, showing which applications have been migrated, which are in progress, and the overall timeline.

Conclusion: Building a Migration-Capable Platform

The deprecation of Ingress-NGINX is a wake-up call. The next major change is a matter of “when,” not “if.” By investing in a structured migration framework now, platform teams transform a potential crisis into a manageable, repeatable operational procedure.

This framework—Assess, Run in Parallel, Validate, and Decommission—abstracts the specific lessons from the ingress migration into a generic pattern. It can be applied to migrating from PodSecurityPolicies to Pod Security Standards, from a deprecated CSI driver, or from one service mesh to another. The tools (GitOps, canary deployments, observability) are already in your stack. The value is in stitching them together into a disciplined process that ensures platform evolution doesn’t compromise platform stability.

Start by documenting this framework as a runbook template. Then, apply it to your next significant component update, even a minor one, to refine the process. When the next major deprecation announcement lands in your inbox, you’ll be ready.

Kubernetes Housekeeping: How to Clean Up Orphaned ConfigMaps and Secrets

Kubernetes Housekeeping: How to Clean Up Orphaned ConfigMaps and Secrets

If you’ve been running Kubernetes clusters for any meaningful amount of time, you’ve likely encountered a familiar problem: orphaned ConfigMaps and Secrets piling up in your namespaces. These abandoned resources don’t just clutter your cluster—they introduce security risks, complicate troubleshooting, and can even impact cluster performance as your resource count grows.

The reality is that Kubernetes doesn’t automatically clean up ConfigMaps and Secrets when the workloads that reference them are deleted. This gap in Kubernetes’ native garbage collection creates a housekeeping problem that every production cluster eventually faces. In this article, we’ll explore why orphaned resources happen, how to detect them, and most importantly, how to implement sustainable cleanup strategies that prevent them from accumulating in the first place.

Understanding the Orphaned Resource Problem

What Are Orphaned ConfigMaps and Secrets?

Orphaned ConfigMaps and Secrets are configuration resources that no longer have any active references from Pods, Deployments, StatefulSets, or other workload resources in your cluster. They typically become orphaned when:

  • Applications are updated and new ConfigMaps are created while old ones remain
  • Deployments are deleted but their associated configuration resources aren’t
  • Failed rollouts leave behind unused configuration versions
  • Development and testing workflows create temporary resources that never get cleaned up
  • CI/CD pipelines generate unique ConfigMap names (often with hash suffixes) on each deployment

Why This Matters for Production Clusters

While a few orphaned ConfigMaps might seem harmless, the problem compounds over time and introduces real operational challenges:

Security Risks: Orphaned Secrets can contain outdated credentials, API keys, or certificates that should no longer be accessible. If these aren’t removed, they remain attack vectors for unauthorized access—especially problematic if RBAC policies grant broad read access to Secrets within a namespace.

Cluster Bloat: Kubernetes stores these resources in etcd, your cluster’s backing store. As the number of orphaned resources grows, etcd size increases, potentially impacting cluster performance and backup times. In extreme cases, this can contribute to etcd performance degradation or even hit storage quotas.

Operational Complexity: When troubleshooting issues or reviewing configurations, sifting through dozens of unused ConfigMaps makes it harder to identify which resources are actually in use. This “configuration noise” slows down incident response and increases cognitive load for your team.

Cost Implications: While individual ConfigMaps are small, at scale they contribute to storage costs and can trigger alerts in cost monitoring systems, especially in multi-tenant environments where resource quotas matter.

Detecting Orphaned ConfigMaps and Secrets

Before you can clean up orphaned resources, you need to identify them. Let’s explore both manual detection methods and automated tooling approaches.

Manual Detection with kubectl

The simplest approach uses kubectl to cross-reference ConfigMaps and Secrets against active workload resources. Here’s a basic script to identify potentially orphaned ConfigMaps:

#!/bin/bash
# detect-orphaned-configmaps.sh
# Identifies ConfigMaps not referenced by any active Pods

NAMESPACE=${1:-default}

echo "Checking for orphaned ConfigMaps in namespace: $NAMESPACE"
echo "---"

# Get all ConfigMaps in the namespace
CONFIGMAPS=$(kubectl get configmaps -n $NAMESPACE -o jsonpath='{.items[*].metadata.name}')

for cm in $CONFIGMAPS; do
    # Skip kube-root-ca.crt as it's system-managed
    if [[ "$cm" == "kube-root-ca.crt" ]]; then
        continue
    fi

    # Check if any Pod references this ConfigMap
    REFERENCED=$(kubectl get pods -n $NAMESPACE -o json | \
        jq -r --arg cm "$cm" '.items[] |
        select(
            (.spec.volumes[]?.configMap.name == $cm) or
            (.spec.containers[].env[]?.valueFrom.configMapKeyRef.name == $cm) or
            (.spec.containers[].envFrom[]?.configMapRef.name == $cm)
        ) | .metadata.name' | head -1)

    if [[ -z "$REFERENCED" ]]; then
        echo "Orphaned: $cm"
    fi
done

A similar script for Secrets would look like this:

#!/bin/bash
# detect-orphaned-secrets.sh

NAMESPACE=${1:-default}

echo "Checking for orphaned Secrets in namespace: $NAMESPACE"
echo "---"

SECRETS=$(kubectl get secrets -n $NAMESPACE -o jsonpath='{.items[*].metadata.name}')

for secret in $SECRETS; do
    # Skip service account tokens and system secrets
    SECRET_TYPE=$(kubectl get secret $secret -n $NAMESPACE -o jsonpath='{.type}')
    if [[ "$SECRET_TYPE" == "kubernetes.io/service-account-token" ]]; then
        continue
    fi

    # Check if any Pod references this Secret
    REFERENCED=$(kubectl get pods -n $NAMESPACE -o json | \
        jq -r --arg secret "$secret" '.items[] |
        select(
            (.spec.volumes[]?.secret.secretName == $secret) or
            (.spec.containers[].env[]?.valueFrom.secretKeyRef.name == $secret) or
            (.spec.containers[].envFrom[]?.secretRef.name == $secret) or
            (.spec.imagePullSecrets[]?.name == $secret)
        ) | .metadata.name' | head -1)

    if [[ -z "$REFERENCED" ]]; then
        echo "Orphaned: $secret"
    fi
done

Important caveat: These scripts only check currently running Pods. They won’t catch ConfigMaps or Secrets referenced by Deployments, StatefulSets, or DaemonSets that might currently have zero replicas. For production use, you’ll want to check against all workload resource types.

Automated Detection with Specialized Tools

Several open-source tools have emerged to solve this problem more comprehensively:

Kor: Comprehensive Unused Resource Detection

Kor is a purpose-built tool for finding unused resources across your Kubernetes cluster. It checks not just ConfigMaps and Secrets, but also PVCs, Services, and other resource types.

# Install Kor
brew install kor

# Scan for unused ConfigMaps and Secrets
kor all --namespace production --output json

# Check specific resource types
kor configmap --namespace production
kor secret --namespace production --exclude-namespaces kube-system,kube-public

Kor works by analyzing resource relationships and identifying anything without dependent objects. It’s particularly effective because it understands Kubernetes resource hierarchies and checks against Deployments, StatefulSets, and DaemonSets—not just running Pods.

Popeye: Cluster Sanitization Reports

Popeye scans your cluster and generates reports on resource health, including orphaned resources. While broader in scope than just ConfigMap cleanup, it provides valuable context:

# Install Popeye
brew install derailed/popeye/popeye

# Scan cluster
popeye --output json --save

# Focus on specific namespace
popeye --namespace production

Custom Controllers with Kubernetes APIs

For more sophisticated detection, you can build custom controllers using client-go that continuously monitor for orphaned resources. This approach works well when integrated with your existing observability stack:

// Pseudocode example
func detectOrphanedConfigMaps(namespace string) []string {
    configMaps := listConfigMaps(namespace)
    deployments := listDeployments(namespace)
    statefulSets := listStatefulSets(namespace)
    daemonSets := listDaemonSets(namespace)

    referenced := make(map[string]bool)

    // Check all workload types for ConfigMap references
    for _, deploy := range deployments {
        for _, cm := range getReferencedConfigMaps(deploy) {
            referenced[cm] = true
        }
    }
    // ... repeat for other workload types

    orphaned := []string{}
    for _, cm := range configMaps {
        if !referenced[cm.Name] {
            orphaned = append(orphaned, cm.Name)
        }
    }

    return orphaned
}

Prevention Strategies: Stop Orphans Before They Start

The best cleanup strategy is prevention. By implementing proper resource management patterns from the beginning, you can minimize orphaned resources in the first place.

Use Owner References for Automatic Cleanup

Kubernetes provides a built-in mechanism for resource lifecycle management through owner references. When properly configured, child resources are automatically deleted when their owner is removed.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
  ownerReferences:
    - apiVersion: apps/v1
      kind: Deployment
      name: myapp
      uid: d9607e19-f88f-11e6-a518-42010a800195
      controller: true
      blockOwnerDeletion: true
data:
  app.properties: |
    database.url=postgres://db:5432

Tools like Helm and Kustomize automatically set owner references, which is one reason GitOps workflows tend to have fewer orphaned resources than imperative deployment approaches.

Implement Consistent Labeling Standards

Labels make it much easier to identify resource relationships and track ownership:

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-gateway-config-v2
  labels:
    app: api-gateway
    component: configuration
    version: v2
    managed-by: argocd
    owner: platform-team
data:
  config.yaml: |
    # configuration here

With consistent labeling, you can easily query for ConfigMaps associated with specific applications:

# Find all ConfigMaps for a specific app
kubectl get configmaps -l app=api-gateway

# Clean up old versions
kubectl delete configmaps -l app=api-gateway,version=v1

Adopt GitOps Practices

GitOps tools like ArgoCD and Flux excel at preventing orphaned resources because they maintain a clear desired state:

  • Declarative management: All resources are defined in Git
  • Automatic pruning: Tools can detect and remove resources not defined in Git
  • Audit trail: Git history shows when and why resources were created or deleted

ArgoCD’s sync policies can automatically prune resources:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  syncPolicy:
    automated:
      prune: true  # Remove resources not in Git
      selfHeal: true

Use Kustomize ConfigMap Generators with Hashes

Kustomize’s ConfigMap generator feature appends content hashes to ConfigMap names, ensuring that configuration changes trigger new ConfigMaps:

# kustomization.yaml
configMapGenerator:
  - name: app-config
    files:
      - config.properties
generatorOptions:
  disableNameSuffixHash: false  # Include hash in name

This creates ConfigMaps like app-config-dk9g72hk5f. When you update the configuration, Kustomize creates a new ConfigMap with a different hash. Combined with Kustomize’s --prune flag, old ConfigMaps are automatically removed:

kubectl apply --prune -k ./overlays/production \
  -l app=myapp

Set Resource Quotas

While quotas don’t prevent orphans, they create backpressure that forces teams to clean up:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: config-quota
  namespace: production
spec:
  hard:
    configmaps: "50"
    secrets: "50"

When teams hit quota limits, they’re incentivized to audit and remove unused resources.

Cleanup Strategies for Existing Orphaned Resources

For clusters that already have accumulated orphaned ConfigMaps and Secrets, here are practical cleanup approaches.

One-Time Manual Cleanup

For immediate cleanup, combine detection scripts with kubectl delete:

# Dry run first - review what would be deleted
./detect-orphaned-configmaps.sh production > orphaned-cms.txt
cat orphaned-cms.txt

# Manual review and cleanup
for cm in $(cat orphaned-cms.txt | grep "Orphaned:" | awk '{print $2}'); do
    kubectl delete configmap $cm -n production
done

Critical warning: Always do a dry run and manual review first. Some ConfigMaps might be referenced by workloads that aren’t currently running but will scale up later (HPA scaled to zero, CronJobs, etc.).

Scheduled Cleanup with CronJobs

For ongoing maintenance, deploy a Kubernetes CronJob that runs cleanup scripts periodically:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: configmap-cleanup
  namespace: kube-system
spec:
  schedule: "0 2 * * 0"  # Weekly at 2 AM Sunday
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: cleanup-sa
          containers:
          - name: cleanup
            image: bitnami/kubectl:latest
            command:
            - /bin/bash
            - -c
            - |
              # Cleanup script here
              echo "Starting ConfigMap cleanup..."

              for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
                echo "Checking namespace: $ns"

                # Get all workload-referenced ConfigMaps
                REFERENCED_CMS=$(kubectl get deploy,sts,ds -n $ns -o json | \
                  jq -r '.items[].spec.template.spec |
                  [.volumes[]?.configMap.name,
                   .containers[].env[]?.valueFrom.configMapKeyRef.name,
                   .containers[].envFrom[]?.configMapRef.name] |
                  .[] | select(. != null)' | sort -u)

                ALL_CMS=$(kubectl get cm -n $ns -o jsonpath='{.items[*].metadata.name}')

                for cm in $ALL_CMS; do
                  if [[ "$cm" == "kube-root-ca.crt" ]]; then
                    continue
                  fi

                  if ! echo "$REFERENCED_CMS" | grep -q "^$cm$"; then
                    echo "Deleting orphaned ConfigMap: $cm in namespace: $ns"
                    kubectl delete cm $cm -n $ns
                  fi
                done
              done
          restartPolicy: OnFailure
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cleanup-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cleanup-role
rules:
- apiGroups: [""]
  resources: ["configmaps", "secrets", "namespaces"]
  verbs: ["get", "list", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets", "daemonsets"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cleanup-binding
subjects:
- kind: ServiceAccount
  name: cleanup-sa
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cleanup-role
  apiGroup: rbac.authorization.k8s.io

Security consideration: This CronJob needs cluster-wide permissions to read workloads and delete ConfigMaps. Review and adjust the RBAC permissions based on your security requirements. Consider limiting to specific namespaces if you don’t need cluster-wide cleanup.

Integration with CI/CD Pipelines

Build cleanup into your deployment workflows. Here’s an example GitLab CI job:

cleanup_old_configs:
  stage: post-deploy
  image: bitnami/kubectl:latest
  script:
    - |
      # Delete ConfigMaps with old version labels after successful deployment
      kubectl delete configmap -n production \
        -l app=myapp,version!=v${CI_COMMIT_TAG}

    - |
      # Keep only the last 3 ConfigMap versions by timestamp
      kubectl get configmap -n production \
        -l app=myapp \
        --sort-by=.metadata.creationTimestamp \
        -o name | head -n -3 | xargs -r kubectl delete -n production
  only:
    - tags
  when: on_success

Safe Deletion Practices

When cleaning up ConfigMaps and Secrets, follow these safety guidelines:

  1. Dry run first: Always review what will be deleted before executing
  2. Backup before deletion: Export resources to YAML files before removing them
  3. Check age: Only delete resources older than a certain threshold (e.g., 30 days)
  4. Exclude system resources: Skip kube-system, kube-public, and other system namespaces
  5. Monitor for impact: Watch application metrics after cleanup to ensure nothing broke

Example backup and conditional deletion:

# Backup before deletion
kubectl get configmap -n production -o yaml > cm-backup-$(date +%Y%m%d).yaml

# Only delete ConfigMaps older than 30 days
kubectl get configmap -n production -o json | \
  jq -r --arg date "$(date -d '30 days ago' -u +%Y-%m-%dT%H:%M:%SZ)" \
  '.items[] | select(.metadata.creationTimestamp < $date) | .metadata.name' | \
  while read cm; do
    echo "Would delete: $cm (created: $(kubectl get cm $cm -n production -o jsonpath='{.metadata.creationTimestamp}'))"
    # Uncomment to actually delete:
    # kubectl delete configmap $cm -n production
  done

Advanced Patterns for Large-Scale Clusters

For organizations running multiple clusters or large multi-tenant platforms, housekeeping requires more sophisticated approaches.

Policy-Based Cleanup with OPA Gatekeeper

Use OPA Gatekeeper to enforce ConfigMap lifecycle policies at admission time:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: configmaprequiredlabels
spec:
  crd:
    spec:
      names:
        kind: ConfigMapRequiredLabels
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package configmaprequiredlabels

        violation[{"msg": msg}] {
          input.review.kind.kind == "ConfigMap"
          not input.review.object.metadata.labels["app"]
          msg := "ConfigMaps must have an 'app' label for lifecycle tracking"
        }

        violation[{"msg": msg}] {
          input.review.kind.kind == "ConfigMap"
          not input.review.object.metadata.labels["owner"]
          msg := "ConfigMaps must have an 'owner' label for lifecycle tracking"
        }

This policy prevents ConfigMaps without proper labels from being created, making future tracking and cleanup much easier.

Centralized Monitoring with Prometheus

Monitor orphaned resource metrics across your clusters:

apiVersion: v1
kind: ConfigMap
metadata:
  name: orphan-detection-exporter
data:
  script.sh: |
    #!/bin/bash
    # Expose metrics for Prometheus scraping
    while true; do
      echo "# HELP k8s_orphaned_configmaps Number of orphaned ConfigMaps"
      echo "# TYPE k8s_orphaned_configmaps gauge"

      for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
        count=$(./detect-orphaned-configmaps.sh $ns | grep -c "Orphaned:")
        echo "k8s_orphaned_configmaps{namespace=\"$ns\"} $count"
      done

      sleep 300  # Update every 5 minutes
    done

Create alerts when orphaned resource counts exceed thresholds:

groups:
- name: kubernetes-housekeeping
  rules:
  - alert: HighOrphanedConfigMapCount
    expr: k8s_orphaned_configmaps > 20
    for: 24h
    labels:
      severity: warning
    annotations:
      summary: "High number of orphaned ConfigMaps in {{ $labels.namespace }}"
      description: "Namespace {{ $labels.namespace }} has {{ $value }} orphaned ConfigMaps"

Multi-Cluster Cleanup with Crossplane or Cluster API

For platform teams managing dozens or hundreds of clusters, extend cleanup automation across your entire fleet:

# Crossplane Composition for cluster-wide cleanup
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: cluster-cleanup-policy
spec:
  compositeTypeRef:
    apiVersion: platform.example.com/v1
    kind: ClusterCleanupPolicy
  resources:
    - name: cleanup-cronjob
      base:
        apiVersion: kubernetes.crossplane.io/v1alpha1
        kind: Object
        spec:
          forProvider:
            manifest:
              apiVersion: batch/v1
              kind: CronJob
              # ... CronJob spec from earlier

Housekeeping Checklist for Production Clusters

Here’s a practical checklist to implement sustainable ConfigMap and Secret housekeeping:

Immediate Actions:

  • [ ] Run detection scripts to audit current orphaned resource count
  • [ ] Backup all ConfigMaps and Secrets before any cleanup
  • [ ] Manually review and delete obvious orphans (with team approval)
  • [ ] Document which ConfigMaps/Secrets are intentionally unused but needed

Short-term (1-4 weeks):

  • [ ] Implement consistent labeling standards across teams
  • [ ] Add owner references to all ConfigMaps and Secrets
  • [ ] Deploy scheduled CronJob for automated detection and reporting
  • [ ] Integrate cleanup steps into CI/CD pipelines

Long-term (1-3 months):

  • [ ] Adopt GitOps tooling (ArgoCD, Flux) with automated pruning
  • [ ] Implement OPA Gatekeeper policies for required labels
  • [ ] Set up Prometheus monitoring for orphaned resource metrics
  • [ ] Create runbooks for incident responders
  • [ ] Establish resource quotas per namespace
  • [ ] Conduct quarterly cluster hygiene reviews

Ongoing Practices:

  • [ ] Review orphaned resource reports weekly
  • [ ] Include cleanup tasks in sprint planning
  • [ ] Train new team members on resource lifecycle best practices
  • [ ] Update cleanup automation as cluster architecture evolves

Conclusion

Kubernetes doesn’t automatically clean up orphaned ConfigMaps and Secrets, but with the right strategies, you can prevent them from becoming a problem. The key is implementing a layered approach: use owner references and GitOps for prevention, deploy automated detection for ongoing monitoring, and run scheduled cleanup jobs for maintenance.

Start with detection to understand your current situation, then focus on prevention strategies like owner references and consistent labeling. For existing clusters with accumulated orphaned resources, implement gradual cleanup with proper safety checks rather than aggressive bulk deletion.

Remember that housekeeping isn’t a one-time task—it’s an ongoing operational practice. By building cleanup into your CI/CD pipelines and establishing clear resource ownership, you’ll maintain a clean, secure, and performant Kubernetes environment over time.

The tools and patterns we’ve covered here—from simple bash scripts to sophisticated policy engines—can be adapted to your organization’s scale and maturity level. Whether you’re managing a single cluster or a multi-cluster platform, investing in proper resource lifecycle management pays dividends in operational efficiency, security posture, and team productivity.

Frequently Asked Questions (FAQ)

Can Kubernetes automatically delete unused ConfigMaps and Secrets?

No. Kubernetes does not garbage-collect ConfigMaps or Secrets by default when workloads are deleted. Unless they have ownerReferences set, these resources remain in the cluster indefinitely and must be cleaned up manually or via automation.

Is it safe to delete ConfigMaps or Secrets that are not referenced by running Pods?

Not always. Some resources may be referenced by workloads scaled to zero, CronJobs, or future rollouts. Always perform a dry run, check workload definitions (Deployments, StatefulSets, DaemonSets), and review resource age before deletion.

What is the safest way to prevent orphaned ConfigMaps and Secrets?

The most effective prevention strategies are:
Using ownerReferences (via Helm or Kustomize)
Adopting GitOps with pruning enabled (ArgoCD / Flux)
Applying consistent labeling (app, owner, version)
These ensure unused resources are automatically detected and removed

Which tools are best for detecting orphaned resources?

Popular and reliable tools include:
Kor – purpose-built for detecting unused Kubernetes resources
Popeye – broader cluster hygiene and sanitization reports
Custom scripts/controllers – useful for tailored environments or integrations
For production clusters, Kor provides the best signal-to-noise ratio.

How often should ConfigMap and Secret cleanup run in production?

A common best practice is:
Weekly detection (reporting only)
Monthly cleanup for resources older than a defined threshold (e.g. 30–60 days)
Immediate cleanup integrated into CI/CD after successful deployments
This balances safety with long-term cluster hygiene.

Sources

Kubernetes Gateway API Versions: Complete Compatibility and Upgrade Guide

Kubernetes Gateway API Versions: Complete Compatibility and Upgrade Guide

The Kubernetes Gateway API has rapidly evolved from its experimental roots to become the standard for ingress and service mesh traffic management. But with multiple versions released and various maturity levels, understanding which version to use, how it relates to your Kubernetes cluster, and when to upgrade can be challenging.

In this comprehensive guide, we’ll explore the different Gateway API versions, their relationship to Kubernetes releases, provider support levels, and the upgrade philosophy that will help you make informed decisions for your infrastructure.

Understanding Gateway API Versioning

The Gateway API follows a unique versioning model that differs from standard Kubernetes APIs. Unlike built-in Kubernetes resources that are tied to specific cluster versions, Gateway API CRDs can be installed independently as long as your cluster meets the minimum requirements.

Minimum Kubernetes Version Requirements

As of Gateway API v1.1 and later versions, you need Kubernetes 1.26 or later to run the latest Gateway API releases. The API commits to supporting a minimum of the most recent 5 Kubernetes minor versions, providing a reasonable window for cluster upgrades.

This rolling support window means that if you’re running Kubernetes 1.26, 1.27, 1.28, 1.29, or 1.30, you can safely install and use the latest Gateway API without concerns about compatibility.

Release Channels: Standard vs Experimental

Gateway API uses two distinct release channels to balance stability with innovation. Understanding these channels is critical for choosing the right version for your use case.

Standard Channel

The Standard channel contains only GA (Generally Available, v1) and Beta (v1beta1) level resources and fields. When you install from the Standard channel, you get:

  • Stability guarantees: No breaking changes once a resource reaches Beta or GA
  • Backwards compatibility: Safe to upgrade between minor versions
  • Production readiness: Extensively tested features with multiple implementations
  • Conformance coverage: Full test coverage ensuring portability

Resources in the Standard channel include GatewayClass, Gateway, HTTPRoute, and ReferenceGrant at the v1 level, plus stable features like GRPCRoute.

Experimental Channel

The Experimental channel includes everything from the Standard channel plus Alpha-level resources and experimental fields. This channel is for:

  • Early feature testing: Try new capabilities before they stabilize
  • Cutting-edge functionality: Access the latest Gateway API innovations
  • No stability guarantees: Breaking changes can occur between releases
  • Feature feedback: Help shape the API by testing experimental features

Features may graduate from Experimental to Standard or be dropped entirely based on implementation experience and community feedback.

Gateway API Version History and Features

Let’s explore the major Gateway API releases and what each introduced.

v1.0 (October 2023)

The v1.0 release marked a significant milestone, graduating core resources to GA status. This release included:

  • Gateway, GatewayClass, and HTTPRoute at v1 (stable)
  • Full backwards compatibility guarantees for v1 resources
  • Production-ready status for ingress traffic management
  • Multiple conformant implementations across vendors

v1.1 (May 2024)

Version 1.1 expanded the API significantly with service mesh support:

  • GRPCRoute: Native support for gRPC traffic routing
  • Service mesh capabilities: East-west traffic management alongside north-south
  • Multiple implementations: Both Istio and other service meshes achieved conformance
  • Enhanced features: Additional matching criteria and routing capabilities

This version bridged the gap between traditional ingress controllers and full service mesh implementations.

v1.2 and v1.3

These intermediate releases introduced structured release cycles and additional features:

  • Refined conformance testing
  • BackendTLSPolicy (experimental in v1.3)
  • Enhanced observability and debugging capabilities
  • Improved cross-namespace routing

v1.4 (October 2025)

The latest GA release as of this writing, v1.4.0 brought:

  • Continued API refinement
  • Additional experimental features for community testing
  • Enhanced conformance profiles
  • Improved documentation and migration guides

Kubernetes Version Compatibility Matrix

Here’s how Gateway API versions relate to Kubernetes releases:

Gateway API Version Minimum Kubernetes Recommended Kubernetes Release Date
v1.0.x 1.25 1.26+ October 2023
v1.1.x 1.26 1.27+ May 2024
v1.2.x 1.26 1.28+ 2024
v1.3.x 1.26 1.29+ 2024
v1.4.x 1.26 1.30+ October 2025

The key takeaway: Gateway API v1.1 and later all support Kubernetes 1.26+, meaning you can run the latest Gateway API on any reasonably modern cluster.

Gateway Provider Support Levels

Different Gateway API implementations support various versions and feature sets. Understanding provider support helps you choose the right implementation for your needs.

Conformance Levels

Gateway API defines three conformance levels for features:

  1. Core: Features that must be supported for an implementation to claim conformance. These are portable across all implementations.
  2. Extended: Standardized optional features. Implementations indicate Extended support separately from Core.
  3. Implementation-specific: Vendor-specific features without conformance requirements.

Major Provider Support

Istio

Istio reached Gateway API GA support in version 1.22 (May 2024). Istio provides:

  • Full Standard channel support (v1 resources)
  • Service mesh (east-west) traffic management via GAMMA
  • Ingress (north-south) traffic control
  • Experimental support for BackendTLSPolicy (Istio 1.26+)

Istio is particularly strong for organizations needing both ingress and service mesh capabilities in a single solution.

Envoy Gateway

Envoy Gateway tracks Gateway API releases closely. Version 1.4.0 includes:

  • Gateway API v1.3.0 support
  • Compatibility matrix for Envoy Proxy versions
  • Focus on ingress use cases
  • Strong experimental feature adoption

Check the Envoy Gateway compatibility matrix to ensure your Envoy Proxy version aligns with your Gateway API and Kubernetes versions.

Cilium

Cilium integrates Gateway API deeply with its CNI implementation:

  • Per-node Envoy proxy architecture
  • Network policy enforcement for Gateway traffic
  • Both ingress and service mesh support
  • eBPF-based packet processing

Cilium’s unique architecture makes it a strong choice for organizations already using Cilium for networking.

Contour

Contour v1.31.0 implements Gateway API v1.2.1, supporting:

  • All Standard channel v1 resources
  • Most v1alpha2 resources (TLSRoute, TCPRoute, GRPCRoute)
  • BackendTLSPolicy support

Checking Provider Conformance

To verify which Gateway API version and features your provider supports:

  1. Visit the official implementations page: The Gateway API project maintains a comprehensive list of implementations with their conformance levels.
  2. Check provider documentation: Most providers publish compatibility matrices showing Gateway API, Kubernetes, and proxy version relationships.
  3. Review conformance reports: Providers submit conformance test results that detail exactly which Core and Extended features they support.
  4. Test in non-production: Before upgrading production, validate your specific use cases in a staging environment.

Upgrade Philosophy: When and How to Upgrade

One of the most common questions about Gateway API is: “Do I need to run the latest version?” The answer depends on your specific needs and risk tolerance.

Staying on Older Versions

You don’t need to always run the latest Gateway API version. It’s perfectly acceptable to:

  • Stay on an older stable release if it meets your needs
  • Upgrade only when you need specific new features
  • Wait for your Gateway provider to officially support newer versions
  • Maintain stability over having the latest features

The Standard channel’s backwards compatibility guarantees mean that when you do upgrade, your existing configurations will continue to work.

When to Consider Upgrading

Consider upgrading when:

  1. You need a specific feature: A new HTTPRoute matcher, GRPCRoute support, or other functionality only available in newer versions
  2. Your provider recommends it: Gateway providers often optimize for specific Gateway API versions
  3. Security considerations: While rare, security issues could prompt upgrades
  4. Kubernetes cluster upgrades: When upgrading Kubernetes, verify your Gateway API version is compatible with the new cluster version

Safe Upgrade Practices

Follow these best practices for Gateway API upgrades:

1. Stick with Standard Channel

Using Standard channel CRDs makes upgrades simpler and safer. Experimental features can introduce breaking changes, while Standard features maintain compatibility.

2. Upgrade One Minor Version at a Time

While it’s usually safe to skip versions, the most tested upgrade path is incremental. Going from v1.2 to v1.3 to v1.4 is safer than jumping directly from v1.2 to v1.4.

3. Test Before Upgrading

Always test upgrades in non-production environments:

# Install specific Gateway API version in test cluster
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

4. Review Release Notes

Each Gateway API release publishes comprehensive release notes detailing:

  • New features and capabilities
  • Graduation of experimental features to standard
  • Deprecation notices
  • Upgrade considerations

5. Check Provider Compatibility

Before upgrading Gateway API CRDs, verify your Gateway provider supports the target version. Installing Gateway API v1.4 won’t help if your controller only supports v1.2.

6. Never Overwrite Different Channels

Implementations should never overwrite Gateway API CRDs that use a different release channel. Keep track of whether you’re using Standard or Experimental channel installations.

CRD Management Best Practices

Gateway API CRD management requires attention to detail:

# Check currently installed Gateway API version
kubectl get crd gateways.gateway.networking.k8s.io -o yaml | grep 'gateway.networking.k8s.io/bundle-version'

# Verify which channel is installed
kubectl get crd gateways.gateway.networking.k8s.io -o yaml | grep 'gateway.networking.k8s.io/channel'

Staying Informed About New Releases

Gateway API releases follow a structured release cycle with clear communication channels.

How to Know When New Versions Are Released

  1. GitHub Releases Page: Watch the kubernetes-sigs/gateway-api repository for release announcements
  2. Kubernetes Blog: Major Gateway API releases are announced on the official Kubernetes blog
  3. Mailing Lists and Slack: Join the Gateway API community channels for discussions and announcements
  4. Provider Announcements: Gateway providers announce support for new Gateway API versions through their own channels

Release Cadence

Gateway API follows a quarterly release schedule for minor versions, with patch releases as needed for bug fixes and security issues. This predictable cadence helps teams plan upgrades.

Practical Decision Framework

Here’s a framework to help you decide which Gateway API version to run:

For New Deployments

  • Production workloads: Use the latest GA version supported by your provider
  • Innovation-focused: Consider Experimental channel if you need cutting-edge features
  • Conservative approach: Use v1.1 or later with Standard channel

For Existing Deployments

  • If things are working: Stay on your current version until you need new features
  • If provider recommends upgrade: Follow provider guidance, especially for security
  • If Kubernetes upgrade planned: Verify compatibility, may need to upgrade Gateway API first or simultaneously

Feature-Driven Upgrades

  • Need service mesh support: Upgrade to v1.1 minimum
  • Need GRPCRoute: Upgrade to v1.1 minimum
  • Need BackendTLSPolicy: Requires v1.3+ and provider support for experimental features

Conclusion

Kubernetes Gateway API represents the future of traffic management in Kubernetes, offering a standardized, extensible, and role-oriented API for both ingress and service mesh use cases. Understanding the versioning model, compatibility requirements, and upgrade philosophy empowers you to make informed decisions that balance innovation with stability.

Key takeaways:

  • Gateway API versions install independently from Kubernetes, requiring only version 1.26 or later for recent releases
  • Standard channel provides stability, Experimental channel provides early access to new features
  • You don’t need to always run the latest version—upgrade when you need specific features
  • Verify provider support before upgrading Gateway API CRDs
  • Follow safe upgrade practices: test first, upgrade incrementally, review release notes

By following these guidelines, you can confidently deploy and maintain Gateway API in your Kubernetes infrastructure while making upgrade decisions that align with your organization’s needs and risk tolerance.

Frequently Asked Questions

What is the difference between Kubernetes Ingress and the Gateway API?

Kubernetes Ingress is a legacy API focused mainly on HTTP(S) traffic with limited extensibility. The Gateway API is its successor, offering a more expressive, role-oriented model that supports multiple protocols, advanced routing, better separation of concerns, and consistent behavior across implementations

Which Gateway API version should I use in production today?

For most production environments, you should use the latest GA (v1.x) release supported by your Gateway provider, installed from the Standard channel. This ensures stability, backwards compatibility, and conformance guarantees while still benefiting from ongoing improvements.

Can I upgrade the Gateway API without upgrading my Kubernetes cluster?

Yes. Gateway API CRDs are installed independently of Kubernetes itself. As long as your cluster meets the minimum supported Kubernetes version (1.26+ for recent releases), you can upgrade the Gateway API without upgrading the cluster.

What happens if my Gateway provider does not support the latest Gateway API version?

If your provider lags behind, you should stay on the latest version officially supported by that provider. Installing newer Gateway API CRDs than your controller supports can lead to missing features or undefined behavior. Provider compatibility should always take precedence over running the newest API version.

Is it safe to upgrade Gateway API CRDs without downtime?

In most cases, yes—when using the Standard channel. The Gateway API provides strong backwards compatibility guarantees for GA and Beta resources. However, you should always test upgrades in a non-production environment and verify that your Gateway provider supports the target version.

Sources

Kubernetes Dashboard Alternatives in 2026: Best Web UI Options After Official Retirement

Kubernetes Dashboard Alternatives in 2026: Best Web UI Options After Official Retirement

The Kubernetes Dashboard, once a staple tool for cluster visualization and management, has been officially archived and is no longer maintained. For many teams who relied on its straightforward web interface to monitor pods, deployments, and services, this retirement marks the end of an era. But it also signals something important: the Kubernetes ecosystem has evolved far beyond what the original dashboard was designed to handle.

Related reading: Headlamp vs FreeLens vs Lens.

Today’s Kubernetes environments are multi-cluster by default, driven by GitOps principles, guarded by strict RBAC policies, and operated by platform teams serving dozens or hundreds of developers. The operating model has simply outgrown the traditional dashboard’s capabilities.

So what comes next? If you’ve been using Kubernetes Dashboard and need to migrate to something more capable, or if you’re simply curious about modern alternatives, this guide will walk you through the best options available in 2026.

Why Kubernetes Dashboard Was Retired

The Kubernetes Dashboard served its purpose well in the early days of Kubernetes adoption. It provided a simple, browser-based interface for viewing cluster resources without needing to master kubectl commands. But as Kubernetes matured, several limitations became apparent:

  • Single-cluster focus: Most organizations now manage multiple clusters across different environments, but the dashboard was designed for viewing one cluster at a time
  • Limited RBAC capabilities: Modern platform teams need fine-grained access controls at the cluster, namespace, and workload levels
  • No GitOps integration: Contemporary workflows rely on declarative configuration and continuous deployment pipelines
  • Minimal observability: Beyond basic resource listing, the dashboard lacked advanced monitoring, alerting, and troubleshooting features
  • Security concerns: The dashboard’s architecture required careful configuration to avoid exposing cluster access

The community recognized these constraints, and the official recommendation now points toward Headlamp as the successor. But Headlamp isn’t the only option worth considering.

Top Kubernetes Dashboard Alternatives for 2026

1. Headlamp: The Official Successor

Headlamp is now the official recommendation from the Kubernetes SIG UI group. It’s a CNCF Sandbox project developed by Kinvolk (now part of Microsoft) that brings a modern approach to cluster visualization.

Key Features:

  • Clean, intuitive interface built with modern web technologies
  • Extensive plugin system for customization
  • Works both as an in-cluster deployment and desktop application
  • Uses your existing kubeconfig file for authentication
  • OpenID Connect (OIDC) support for enterprise SSO
  • Read and write operations based on RBAC permissions

Installation Options:

# Using Helm
helm repo add headlamp https://kubernetes-sigs.github.io/headlamp/
helm install my-headlamp headlamp/headlamp --namespace kube-system

# As Minikube addon
minikube addons enable headlamp
minikube service headlamp -n headlamp

Headlamp excels at providing a familiar dashboard experience while being extensible enough to grow with your needs. The plugin architecture means you can customize it for your specific workflows without waiting for upstream changes.

Best for: Teams transitioning from Kubernetes Dashboard who want a similar experience with modern features and official backing.

2. Portainer: Enterprise Multi-Cluster Management

Portainer has evolved from a Docker management tool into a comprehensive Kubernetes platform. It’s particularly strong when you need to manage multiple clusters from a single interface. We already covered in detail Portainer so you can also take a look

Key Features:

  • Multi-cluster management dashboard
  • Enterprise-grade RBAC with fine-grained access controls
  • Visual workload deployment and scaling
  • GitOps integration support
  • Comprehensive audit logging
  • Support for both Kubernetes and Docker environments

Best for: Organizations managing multiple clusters across different environments who need enterprise RBAC and centralized control.

3. Skooner (formerly K8Dash): Lightweight and Fast

Skooner keeps things simple. If you appreciated the straightforward nature of the original Kubernetes Dashboard, Skooner delivers a similar philosophy with a cleaner, faster interface.

Key Features:

  • Fast, real-time updates
  • Clean and minimal interface
  • Easy installation with minimal configuration
  • Real-time metrics visualization
  • Built-in OIDC authentication

Best for: Teams that want a simple, no-frills dashboard without complex features or steep learning curves.

4. Devtron: Complete DevOps Platform

Devtron goes beyond simple cluster visualization to provide an entire application delivery platform built on Kubernetes.

Key Features:

  • Multi-cluster application deployment
  • Built-in CI/CD pipelines
  • Advanced security scanning and compliance
  • Application-centric view rather than resource-centric
  • Support for seven different SSO providers
  • Chart store for Helm deployments

Best for: Platform teams building internal developer platforms who need comprehensive deployment pipelines alongside cluster management.

5. KubeSphere: Full-Stack Container Platform

KubeSphere positions itself as a distributed operating system for cloud-native applications, using Kubernetes as its kernel.

Key Features:

  • Multi-tenant architecture
  • Integrated DevOps workflows
  • Service mesh integration (Istio)
  • Multi-cluster federation
  • Observability and monitoring built-in
  • Plug-and-play architecture for third-party integrations

Best for: Organizations building comprehensive container platforms who want an opinionated, batteries-included experience.

6. Rancher: Battle-Tested Enterprise Platform

Rancher from SUSE has been in the Kubernetes management space for years and offers one of the most mature platforms available.

Key Features:

  • Manage any Kubernetes cluster (EKS, GKE, AKS, on-premises)
  • Centralized authentication and RBAC
  • Built-in monitoring with Prometheus and Grafana
  • Application catalog with Helm charts
  • Policy management and security scanning

Best for: Enterprise organizations managing heterogeneous Kubernetes environments across multiple cloud providers.

7. Octant: Developer-Focused Cluster Exploration

Octant (originally developed by VMware) takes a developer-centric approach to cluster visualization with a focus on understanding application architecture.

Key Features:

  • Plugin-based extensibility
  • Resource relationship visualization
  • Port forwarding directly from the UI
  • Log streaming
  • Context-aware resource inspection

Best for: Application developers who need to understand how their applications run on Kubernetes without being cluster administrators.

Desktop and CLI Alternatives Worth Considering

While this article focuses on web-based dashboards, it’s worth noting that not everyone needs a browser interface. Some of the most powerful Kubernetes management tools work as desktop applications or terminal UIs.

If you’re considering client-side tools, you might find these articles on my blog helpful:

These client tools offer advantages that web dashboards can’t match: offline access, better performance, and tighter integration with your local development workflow. FreeLens, in particular, has emerged as the lowest-risk choice for most organizations looking for a desktop Kubernetes IDE.

Choosing the Right Alternative for Your Team

With so many options available, how do you choose? Here’s a decision framework:

Choose Headlamp if:

  • You want the officially recommended path forward
  • You need a lightweight dashboard similar to what you had before
  • Plugin extensibility is important for future customization
  • You prefer CNCF-backed open source projects

Choose Portainer if:

  • You manage multiple Kubernetes clusters
  • Enterprise RBAC is a critical requirement
  • You also work with Docker environments
  • Visual deployment tools would benefit your team

Choose Skooner if:

  • You want the simplest possible alternative
  • Your needs are straightforward: view and manage resources
  • You don’t need advanced features or multi-cluster support

Choose Devtron or KubeSphere if:

  • You’re building an internal developer platform
  • You need integrated CI/CD pipelines
  • Application-centric workflows matter more than resource-centric views

Choose Rancher if:

  • You’re managing enterprise-scale, multi-cloud Kubernetes
  • You need battle-tested stability and vendor support
  • Policy management and compliance are critical

Consider desktop tools like FreeLens if:

  • You work primarily from a local development environment
  • You need offline access to cluster information
  • You prefer richer desktop application experiences

Migration Considerations

If you’re actively using Kubernetes Dashboard today, here’s what to think about when migrating:

  1. Authentication method: Most modern alternatives support OIDC/SSO, but verify your specific identity provider is supported
  2. RBAC policies: Review your existing ClusterRole and RoleBinding configurations to ensure they translate properly
  3. Custom workflows: If you’ve built automation around Dashboard URLs or specific features, you’ll need to adapt these
  4. User training: Even similar-looking alternatives have different UIs and workflows; budget time for team training
  5. Ingress configuration: If you expose your dashboard externally, you’ll need to reconfigure ingress rules

The Future of Kubernetes UI Management

The retirement of Kubernetes Dashboard isn’t a step backward—it’s recognition that the ecosystem has matured. Modern platforms need to handle multi-cluster management, GitOps workflows, comprehensive observability, and sophisticated RBAC out of the box.

The alternatives listed here represent different philosophies about what a Kubernetes interface should be:

  • Minimalist dashboards (Headlamp, Skooner) that stay close to the original vision
  • Enterprise platforms (Portainer, Rancher) that centralize multi-cluster management
  • Developer platforms (Devtron, KubeSphere) that integrate the entire application lifecycle
  • Desktop experiences (FreeLens, OpenLens) that bring IDE-like capabilities

The right choice depends on your team’s size, your infrastructure complexity, and whether you’re managing platforms or building applications. For most teams migrating from Kubernetes Dashboard, starting with Headlamp makes sense—it’s officially recommended, actively maintained, and provides a familiar experience. From there, you can evaluate whether you need to scale up to more comprehensive platforms.

Whatever you choose, the good news is that the Kubernetes ecosystem in 2026 offers more sophisticated, capable, and secure dashboard alternatives than ever before.

Frequently Asked Questions (FAQ)

Is Kubernetes Dashboard officially deprecated or just unmaintained?

u003cbru003eThe Kubernetes Dashboard has been officially u003cstrongu003earchivedu003c/strongu003e by the Kubernetes project and is no longer actively maintained. While it may still run in existing clusters, it no longer receives security updates, bug fixes, or new features, making it unsuitable for production use in modern environments.

What is the official replacement for Kubernetes Dashboard?

u003cstrongu003eHeadlampu003c/strongu003e is the officially recommended successor by the Kubernetes SIG UI group. It provides a modern web interface, supports plugins, integrates with existing kubeconfig files, and aligns with current Kubernetes security and RBAC best practices.

Is Headlamp production-ready for enterprise environments?

Yes. Headlamp supports u003cstrongu003eOIDC authenticationu003c/strongu003e, fine-grained u003cstrongu003eRBACu003c/strongu003e, and can run either in-cluster or as a desktop application. While still evolving, it is actively maintained and suitable for many production use cases, especially when combined with proper access controls.

Are there lightweight alternatives similar to the old Kubernetes Dashboard?

Yes. u003cstrongu003eSkooneru003c/strongu003e is a lightweight, fast alternative that closely mirrors the simplicity of the original Kubernetes Dashboard while offering a cleaner UI and modern authentication options like OIDC.

Do I still need a web-based dashboard to manage Kubernetes?

Not necessarily. Many teams prefer u003cstrongu003edesktop or CLI-based toolsu003c/strongu003e such as FreeLens, OpenLens, or K9s. These tools often provide better performance, offline access, and deeper integration with developer workflows compared to browser-based dashboards.

Is it safe to expose Kubernetes dashboards over the internet?

Exposing any Kubernetes dashboard publicly requires u003cstrongu003eextreme cautionu003c/strongu003e. If external access is necessary, always use:u003cbru003eStrong authentication (OIDC / SSO)u003cbru003eStrict RBAC policiesu003cbru003eNetwork restrictions (VPN, IP allowlists)u003cbru003eTLS termination and hardened ingress rulesu003cbru003eIn many cases, dashboards should only be accessible from internal networks.

Can these dashboards replace kubectl?

No. Dashboards are u003cstrongu003ecomplementary toolsu003c/strongu003e, not replacements for u003ccodeu003ekubectlu003c/codeu003e. While they simplify visualization and some management tasks, advanced operations, automation, and troubleshooting still rely heavily on CLI tools and GitOps workflows.

What should I consider before migrating away from Kubernetes Dashboard?

Before migrating, review:u003cbru003eAuthentication and identity provider compatibilityu003cbru003eExisting RBAC roles and permissionsu003cbru003eMulti-cluster requirementsu003cbru003eGitOps and CI/CD integrationsu003cbru003eTraining needs for platform teams and developersu003cbru003eStarting with Headlamp is often the lowest-risk migration path

Which Kubernetes dashboard is best for developers rather than platform teams?

Tools like u003cstrongu003eOctantu003c/strongu003e and u003cstrongu003eDevtronu003c/strongu003e are more developer-focused. They emphasize application-centric views, resource relationships, and deployment workflows, making them ideal for developers who want insight without managing cluster infrastructure directly.

Which Kubernetes dashboard is best for multi-cluster management?

For multi-cluster environments, u003cstrongu003ePortaineru003c/strongu003e, u003cstrongu003eRancheru003c/strongu003e, and u003cstrongu003eKubeSphereu003c/strongu003e are strong options. These platforms are designed to manage multiple clusters from a single control plane and offer enterprise-grade RBAC, auditing, and centralized authentication.

Related reading: FreeLens extensions: the complete catalogue and how to install them.

FreeLens vs OpenLens vs Lens: Choosing the Right Kubernetes IDE

FreeLens vs OpenLens vs Lens: Choosing the Right Kubernetes IDE

Introduction: When a Tool Choice Becomes a Legal and Platform Decision

If you’ve been operating Kubernetes clusters for a while, you’ve probably learned this the hard way:
tooling decisions don’t stay “just tooling” for long.

Related reading: how Headlamp compares to FreeLens and Lens.

And if you landed here because the built-in Kubernetes Dashboard was retired and you’re weighing your alternatives, these desktop IDEs are only half the picture — browser-based dashboards are the other half.

What starts as a developer convenience can quickly turn into:

  • a licensing discussion with Legal,
  • a procurement problem,
  • or a platform standard you’re stuck with for years.

The Kubernetes IDE ecosystem is a textbook example of this.

Many teams adopted Lens because it genuinely improved day-to-day operations. Then the license changed. Then restrictions appeared. Then forks started to emerge.

Today, the real question is not “Which one looks nicer?” but:

  • Which one is actually maintained?
  • Which one is safe to use in a company?
  • Why is there a fork of a fork?
  • Are they still technically compatible?
  • What is the real switch cost?

Let’s go through this from a production and platform engineering perspective.

What Is FreeLens?

FreeLens is a free, open-source desktop IDE for Kubernetes, released under the MIT license. It is a community fork of OpenLens — which was itself the open-source core of Lens Desktop — created in 2024 to keep a genuinely open version of the tool alive after Mirantis restricted the Lens binaries and the OpenLens repository went dormant. It gives you the same thing people originally loved about Lens: a cluster list, a live view of your workloads, logs, a terminal, and resource editing, all from a desktop app that reads your existing kubeconfig.

As of this writing the current release is FreeLens v1.10.3 (July 2026), the project has roughly 5.3k GitHub stars, and it ships installers for macOS, Windows and Linux on both amd64 and arm64. Unlike Lens Desktop, there is no account, no sign-in, and no commercial licence to negotiate — you download it and it works.

How to Install FreeLens

FreeLens is packaged for every mainstream package manager, so in most cases installation is a single command:

# macOS (Homebrew)
brew install --cask freelens

# Windows (WinGet)
winget install Freelensapp.Freelens

# Linux (Snap)
snap install freelens --classic

# Linux (Flatpak)
flatpak install flathub app.freelens.Freelens
flatpak run app.freelens.Freelens

On Debian and Ubuntu you can add the project’s APT repository instead, which gets you upgrades through your normal apt upgrade cycle:

curl -L https://raw.githubusercontent.com/freelensapp/freelens/refs/heads/main/freelens/build/apt/freelens.asc 
  | sudo tee /etc/apt/keyrings/freelens.asc
curl -L https://raw.githubusercontent.com/freelensapp/freelens/refs/heads/main/freelens/build/apt/freelens.sources 
  | sudo tee /etc/apt/sources.list.d/freelens.sources
sudo apt update
sudo apt install freelens

If you would rather not use a package manager, the GitHub releases page carries direct installers for every platform: .dmg and .pkg for macOS, .exe, .msi and a portable .exe for Windows, and .AppImage, .deb and .rpm for Linux — all built for both amd64 and arm64, and all shipped with SHA256 checksums and an SBOM, which is a detail your security team will appreciate more than you expect.

Once installed, FreeLens picks up ~/.kube/config automatically. If you manage several clusters, everything you already do with contexts keeps working — there is no separate cluster registration step.

FreeLens Extensions

Extensions are the main reason people stayed on the Lens family rather than moving to a browser dashboard, and this is the area where FreeLens has done the most catching up. The project maintains its own extension ecosystem under the freelensapp GitHub organisation, and many extensions originally written for OpenLens have already been ported. The ones worth knowing about:

  • freelens-fluxcd-extension — by far the most adopted one (~123 stars). Surfaces FluxCD resources (Kustomizations, HelmReleases, sources) as first-class objects in the UI. If you run GitOps with Flux, this is the one to install first.
  • freelens-ai-extension — AI-assisted cluster interrogation from inside the IDE.
  • freelens-gateway-api-extension — views for Gateway API objects (Gateways, HTTPRoutes), which the base UI still treats as generic CRDs. Pairs well with our Gateway API version compatibility guide.
  • freelens-karpenter-extension — inspect NodePools and Karpenter-provisioned nodes without dropping to the CLI. Useful alongside Cluster Autoscaler vs Karpenter.
  • freelens-kamaji-extension and freelens-sveltos-extension — for hosted control planes (Kamaji) and multi-cluster add-on distribution (Sveltos).
  • freelens-example-extension — the scaffold to start from if you want to write your own. The extension API is inherited from Lens, so older OpenLens extension code usually ports with modest changes.

The full catalogue — official and community extensions with their exact npm package names, plus the ones that no longer do anything — is in FreeLens extensions: the complete catalogue and how to install them.

One caveat worth flagging: freelens-node-pod-menu is deprecated — its node and pod context-menu actions were folded into the core application, so you do not need it on current versions. If you are following an older tutorial that tells you to install it, skip that step.

What Is OpenLens?

OpenLens was the open-source core of Lens Desktop — the repository that held the actual application code, minus the proprietary extras Mirantis layered on top of the distributed binaries. For a couple of years it was the standard answer to “how do I use Lens without agreeing to the commercial terms”: you built OpenLens yourself, or you used one of the community builds, and you got the Lens experience without the account requirement.

That era is over. OpenLens is no longer maintained, and community builds stopped tracking upstream. If you are running it today you are running unpatched Electron and unpatched dependencies, which is the part that turns a tooling preference into a security conversation. The practical migration path is FreeLens — it is the direct continuation of the same codebase, so your kubeconfig, your contexts and most of your extensions carry over. The detail on what breaks and what does not is in the migration section further down.

FreeLens vs OpenLens vs Lens: Side-by-Side

The whole decision in one table. The detail behind each row is in the sections that follow.

FreeLensOpenLensLens Desktop
Status✅ Actively maintained❌ Abandoned✅ Commercial product
LicenceMITMIT (unmaintained)Proprietary (Mirantis)
Current releasev1.10.3 (Jul 2026)— no longer releasedRolling
Account requiredNoNoYes
Free for commercial use✅ Yes✅ Yes, but unpatched❌ Requires a licence
Security updates✅ Current❌ None — unpatched Electron✅ Vendor-maintained
ExtensionsOwn ecosystem; most OpenLens ones portedLegacy OpenLens extensionsCommercial catalogue
Installbrew, winget, snap, flatpak, apt, directCommunity builds onlyVendor installer
PlatformsmacOS, Windows, Linux (amd64 + arm64)Varies by buildmacOS, Windows, Linux
Verdict✅ Default choice in 2026❌ Migrate off itOnly if you already pay for it

There is also a fourth option that is not part of this lineage at all: Headlamp, a CNCF-governed Kubernetes UI and the only one of these that runs as a shared in-cluster web UI with access controlled by Kubernetes RBAC rather than by each engineer’s local kubeconfig.

The Forking Story: How We Ended Up Here

Understanding the lineage matters because it explains why FreeLens exists at all.

Lens: The Original Product

Lens started as an open-core Kubernetes IDE with a strong community following. Over time, it evolved into a commercial product with:

  • a proprietary license,
  • paid enterprise features,
  • and restrictions on free usage in corporate environments.

This shift was legitimate from a business perspective, but it broke the implicit contract many teams assumed when they standardized on it.

OpenLens: The First Fork

OpenLens was created to preserve:

  • open-source licensing,
  • unrestricted commercial usage,
  • compatibility with Lens extensions.

For a while, OpenLens was the obvious alternative for teams that wanted to stay open-source without losing functionality.

FreeLens: The Fork of the Fork

FreeLens appeared later, and this is where many people raise an eyebrow.

Why fork OpenLens?

Because OpenLens development started to slow down:

  • release cadence became irregular,
  • upstream Kubernetes changes lagged,
  • governance and long-term stewardship became unclear.

FreeLens exists because some contributors were not willing to bet their daily production tooling on a project with uncertain momentum.

This was not ideology. It was operational risk management.

Are the Projects Still Maintained?

Short answer: yes, but not equally.

Lens

  • Actively developed
  • Backed by a commercial vendor
  • Fast adoption of new Kubernetes features

Trade-off:

  • Licensing constraints
  • Paid features
  • Requires legal review in most companies

OpenLens

  • Still maintained
  • Smaller contributor base
  • Slower release velocity

It works, but it no longer feels like a safe long-term default for platform teams.

FreeLens

  • Actively maintained
  • Explicit focus on long-term openness
  • Prioritizes Kubernetes API compatibility and stability

Right now, FreeLens shows the healthiest balance between maintenance and independence.

Technical Compatibility: Can You Switch Without Pain?

This is the good news: yes, mostly.

Cluster Access and Configuration

All three tools:

  • use standard kubeconfig files,
  • support multiple contexts and clusters,
  • work with RBAC, CRDs, and namespaces the same way.

No cluster-side changes are required.

Extensions and Plugins

  • Most Lens extensions work in OpenLens.
  • Most OpenLens extensions work in FreeLens.
  • Proprietary Lens-only extensions are the main exception.

In real-world usage:

  • ~90% of common workflows are identical
  • differences show up only in edge cases or paid features

UX Differences

There are some UI differences:

  • branding,
  • menu structure,
  • feature gating in Lens.

Nothing that requires retraining or documentation updates.

Legal and Licensing Considerations (This Is Where It Usually Breaks)

This is often the decisive factor in enterprise environments.

Lens

  • Requires license compliance checks
  • Free usage may violate internal policies
  • Paid plans required for broader adoption

If you operate in a regulated or audited environment, this alone can be a blocker.

OpenLens

  • Open-source license
  • Generally safe for corporate use
  • Slight uncertainty due to reduced activity

FreeLens

  • Explicitly open-source
  • No usage restrictions
  • Clear intent to remain free for commercial use

If Legal asks, “Can we standardize this across the company?”
FreeLens is the easiest answer.

OpenLens Is Abandoned: What to Use Instead (and How to Migrate)

If you landed here searching for an OpenLens alternative: OpenLens is effectively abandoned. The binary-build repository shipped its last release (v6.5.2) in June 2023 and explicitly warns that no more updates are coming, because upstream Lens closed its source code. The repo is not formally archived, but the practical result is the same — no features, no bug fixes, and critically, no security patches for an application that holds credentials to every cluster you manage.

The migration path is FreeLens, the actively maintained community fork. The switch is low-friction:

  • Install FreeLens (Homebrew, winget, or binaries) — it reads the same kubeconfig files, so your clusters appear immediately.
  • Reinstall your extensions from the FreeLens extension ecosystem; the most-used OpenLens extensions have equivalents.
  • Uninstall OpenLens once you have verified access — leaving an unpatched credential-holding app installed is the real risk.

Which One Should You Use in a Company?

A pragmatic recommendation:

Use Lens if:

  • you want vendor-backed support,
  • you are willing to pay,
  • you already standardized on Mirantis tooling.

Use OpenLens if:

  • you are already using it,
  • it meets your needs today,
  • you accept slower updates.

Use FreeLens if:

  • you want zero licensing risk,
  • you want an open-source default,
  • you care about long-term maintenance,
  • you need something you can standardize safely.

For most platform and DevOps teams, FreeLens is currently the lowest-risk choice.

Switch Cost: How Expensive Is It Really?

Surprisingly low.

Typical migration:

  • install the new binary,
  • reuse existing kubeconfigs,
  • reinstall extensions if needed.

What you don’t need:

  • cluster changes,
  • CI/CD modifications,
  • platform refactoring.

Downtime: none
Rollback: trivial

This is one of the rare cases where switching early is cheap.

Is a “Fork of a Fork” a Red Flag?

Normally, yes.

In this case, no.

FreeLens exists because:

  • maintenance mattered more than branding,
  • openness mattered more than monetization,
  • predictability mattered more than roadmap promises.

Ironically, this is very aligned with how Kubernetes itself evolved.

Frequently Asked Questions

Is OpenLens still maintained?

No. The OpenLens binary-build repository shipped its last release (v6.5.2) in June 2023 and states that no further updates should be expected, since upstream Lens closed its source code. It receives no features, fixes or security patches.

What is the best OpenLens alternative in 2026?

FreeLens is the natural replacement: an actively maintained community fork with the same core UX, an extension ecosystem, and regular releases. It reads your existing kubeconfig files, so migration takes minutes.

Can I migrate from OpenLens to FreeLens without losing my clusters?

Yes. Both tools discover clusters from your kubeconfig files, so FreeLens picks up the same clusters automatically. Extensions need to be reinstalled from the FreeLens catalog.

Is Lens free for commercial use?

Lens (the commercial product by Mirantis) requires a subscription for most business use; check current licensing terms. This is one of the main reasons teams standardize on FreeLens, which is open source under the MIT license.

Conclusion: A Clear, Boring, Production-Safe Answer

If you strip away GitHub drama and branding:

  • Lens optimizes for revenue and enterprise features.
  • OpenLens preserved openness but lost momentum.
  • FreeLens optimizes for sustainability and freedom.

From a platform engineering perspective:

FreeLens is the safest default Kubernetes IDE today for most organizations.

Low switch cost, strong compatibility, no legal surprises.

And in production environments, boring and predictable almost always wins.

One last platform note: these IDEs only manage the cluster — they don’t change what runs underneath it. If you’re also rethinking the node OS, our Talos Linux guide covers the immutable, API-driven base many teams now pair with tools like FreeLens.