If you manage Kubernetes in production, you’ve likely felt the sting of a feature that promised stability but delivered chaos. The community is rich with stories of PodDisruptionBudgets (PDBs) that blocked critical updates, misconfigured liveness probes that created restart loops, and resource limits that turned a simple deployment into a cascading failure. These aren’t inherently “bad” features; they are powerful tools that, like a surgeon’s scalpel, require precise understanding and context to use effectively.
Related reading: liveness probe anti-patterns.
Drawing from a wealth of shared experience—including community discussions and documented failure stories—a clear pattern emerges. The gap between a feature’s theoretical promise and its production reality is often bridged not by more documentation, but by operational rigor. This post analyzes common pitfalls, not to discourage the use of these features, but to provide a decision framework for platform teams to evaluate adoption, focusing on observability, gradual rollout, and clear rollback plans.
The Gap Between Theory and Practice: Features That Bite Back
Kubernetes is designed to automate complex distributed systems patterns. However, this automation can amplify misconfigurations at scale. The following features are frequently cited as sources of production pain, precisely because their power is double-edged.
1. PodDisruptionBudgets (PDBs): The Update Blocker
On paper, a PDB is a safeguard. It ensures a minimum number of pods for a critical application remain available during voluntary disruptions like node drains or cluster upgrades. The theory is flawless.
The practice, as shared by many engineers, reveals the trap: a PDB with overly restrictive minAvailable or maxUnavailable settings can completely halt cluster maintenance. Imagine a deployment with 3 pods and a PDB set to minAvailable: 3. Any drain operation is now impossible, stalling node security patches or Kubernetes version upgrades. The cluster’s ability to heal and evolve is held hostage by a configuration intended to protect it.
The deeper lesson isn’t to avoid PDBs, but to configure them with the system’s evolution in mind. They must allow for the cluster’s own lifecycle operations.
2. Liveness and Readiness Probes: The Self-Inflicted Outage
Probes are the cornerstone of Kubernetes’ self-healing and traffic management. A liveness probe failure restarts the pod; a readiness probe failure removes it from service endpoints. This is essential for resilience.
In production, misconfigured probes are a classic source of instability. Common pitfalls include:
- Overly sensitive liveness checks: A probe checking an endpoint that briefly spikes in latency due to a downstream cache miss can cause a restart loop, exacerbating the problem and taking the service fully down.
- Resource-intensive probes: A probe that executes a heavy database query every few seconds can itself become a source of resource exhaustion and latency, creating a feedback loop of failure.
- Incorrect readiness signals: An application marked “not ready” during its entire startup or lengthy initialization will never receive traffic, appearing as a deployment failure.
As noted in the Kubernetes configuration overview, probes must be designed to reflect the actual health of the application, not an idealized state. They should be cheap, stable, and representative.
3. Resource Requests and Limits: The Silent Strangulation
Setting CPU and memory requests/limits is Kubernetes 101. They ensure fair scheduling and prevent a single pod from consuming all node resources. The theory is fundamental to multi-tenancy.
The production reality is subtler. Setting limits too low (“limit starvation”) is a frequent cause of mysterious, intermittent failures. A pod hitting its CPU limit is throttled, causing increased latency and timeouts. A pod hitting its memory limit is OOMKilled instantly. The symptoms—slow responses or disappearing pods—often point to application bugs, masking the true infrastructure cause.
Conversely, setting requests too high leads to poor cluster utilization and scheduling headaches. The key is continuous observation: limits should be informed by actual usage under load, not initial guesses.
4. Helm Hooks and Complex Operators: The Unpredictable Orchestrator
Helm hooks and custom operators automate complex lifecycle tasks: database migrations, secret injection, or pre-upgrade validation. They abstract away imperative steps.
In production, this abstraction can become a black box. A post-install hook that fails can leave a release in a stuck state. An operator with a bug in its reconciliation logic can enter a loop, endlessly creating and deleting resources. The complexity of debugging an automated system that has gone awry often far exceeds the complexity of the manual process it replaced. The failure stories aggregated in resources like kubernetes-failure-stories are replete with examples of automation gone wrong.
A Framework for Safer Feature Adoption
Banning powerful features is not the answer. The goal is to adopt them with eyes wide open. Platform teams should implement a framework that evaluates risk and mandates safeguards. Here is a practical, four-phase approach.
Phase 1: Evaluation & Contextual Understanding
Before enabling a feature cluster-wide or recommending it to application teams, ask:
- What problem does this solve for us? Is it a real pain point, or just a “nice-to-have”?
- What is the failure mode? How can this feature break? (e.g., PDBs block drains, probes cause restarts).
- What are the observability requirements? What metrics, logs, and alerts do we need to see if it’s misbehaving?
- What is the rollback procedure? How do we quickly disable or revert this feature if it causes an incident?
Phase 2: Implementation with Guardrails
Deploy the feature with constraints that limit its blast radius.
- Start with non-critical workloads: Apply PDBs first to staging or low-priority services.
- Use sane defaults via Policy-as-Code: Use tools like OPA/Gatekeeper or Kyverno to enforce safe defaults. For example, a policy could forbid PDBs with
minAvailable: 100%or enforce a maximum probe timeout. - Document the “why” and the “how to escape”: Annotate resources or maintain runbooks that explain the configuration and the steps to neutralize it in an emergency.
Phase 3: Gradual Rollout & Observability
Treat feature adoption like a software deployment.
- Canary the configuration: Apply a new PDB or aggressive probe to one pod or one namespace first. Monitor its effect closely.
- Implement specific monitoring: Beyond general cluster health, create alerts for:
– PDBs blocking evictions for > X minutes.
– Pod restart counts spiking (potential probe issue).
– Containers hitting CPU throttling or being OOMKilled.
– Helm releases stuck in a pending hook state.
A simple Prometheus alert for PDB blockage might look like this:
# Alert if a PDB is blocking voluntary pod disruptions for too long
- alert: PDBBlockingDisruption
expr: kube_poddisruptionbudget_status_current_healthy == kube_poddisruptionbudget_status_desired_healthy
and (kube_poddisruptionbudget_status_desired_healthy - kube_poddisruptionbudget_status_expected_pods) == 0
and kube_poddisruptionbudget_status_disruptions_allowed == 0
for: 10m
labels:
severity: warning
annotations:
summary: "PDB {{ $labels.namespace }}/{{ $labels.poddisruptionbudget }} is blocking all pod disruptions"
description: "The PDB requires all pods to be available, preventing node drains or updates for 10 minutes."
Phase 4: Review and Iteration
Adoption isn’t a one-time event. Regularly review:
- Are the features providing the intended value? Are PDBs actually increasing availability during updates?
- What incidents or near-misses have they been involved in? Use post-incident reviews to refine configurations and policies.
- Can we improve defaults or abstractions? Can the platform team provide a simplified, safe Custom Resource or Helm chart that encapsulates best practices?
Frequently Asked Questions
Which Kubernetes features cause the most production incidents?
The recurring offenders are the ones that act automatically on your behalf: PodDisruptionBudgets that block node drains forever, liveness probes that restart healthy pods under load, and aggressive affinity rules that make workloads unschedulable. None of them are bad features u2014 they hurt when adopted with defaults copied from a tutorial instead of settings derived from your workload.
Why is a PodDisruptionBudget risky if it protects availability?
Because a PDB with maxUnavailable: 0 (or a selector matching a single replica) makes voluntary disruption impossible: node drains hang, upgrades stall, and cluster maintenance turns into manual pod deletion at 2 AM. A PDB must always leave the cluster a legal way to move your pods.
Should I avoid these features altogether?
No u2014 the article’s point is the opposite. Adopt them deliberately: understand the failure mode each feature introduces, test that failure mode (drain a node, kill a replica) before production, and roll out with conservative settings you tighten over time. Power tools, respected.
How do I evaluate a Kubernetes feature before adopting it?
Four questions: What does it do automatically and when? What is the failure mode when it misfires u2014 and does it fail open or closed? Can I observe it acting (events, metrics)? And can I roll it back under pressure? If you cannot answer all four, you are adopting a behavior, not a feature.
Conclusion: Embrace Power, Respect Complexity
The history of engineering is the history of building more powerful tools and learning to wield them safely. Kubernetes features like PDBs, probes, and resource management are no different. Their potential for causing production pain is a direct reflection of their power to automate complex, critical behaviors.
The path forward is not avoidance, but disciplined adoption. By shifting from a mindset of “enable and hope” to a framework of “evaluate, guard, observe, and iterate,” platform teams can harness these powerful features to build more resilient, self-healing systems without becoming victims of their own automation. The lessons are already written in the community’s failure stories; the task is to learn from them and build a safer, more informed practice.