Beyond GPU Drivers: Custom Node Readiness for Specialized Workloads

If you manage Kubernetes clusters for specialized workloads—think AI/ML, high-performance computing, financial modeling, or real-time data processing—you know that a node being Ready in the Kubernetes sense is often just the starting point. The standard kubelet health checks ensure the node’s core services are running, but they say nothing about whether the specific, often expensive, hardware or software dependencies your workload requires are truly operational.

Related reading: GPU scheduling with Dynamic Resource Allocation.

You might have nodes with GPUs where the driver crashed, nodes with local NVMe volumes that haven’t finished formatting, or nodes where a mandatory security or monitoring agent is unhealthy. Scheduling a sensitive, resource-intensive pod onto such a node is a recipe for silent failures, degraded performance, or security non-compliance.

This is where the Node Readiness Controller moves from a convenience to a critical platform control plane component. While its introductory examples often focus on GPU driver checks, its real power lies in enforcing your platform’s unique definition of “ready.” In this article, we’ll move beyond the basics and explore how to implement custom node readiness gates for advanced, production-grade scenarios.

Why Standard Node Readiness Isn’t Enough

Kubernetes marks a node as Ready when its kubelet can communicate with the API server and reports that essential node conditions (like MemoryPressure, DiskPressure, PIDPressure, and the generic Ready) are false. This is a binary, infrastructure-level view.

For specialized workloads, you need to enforce application-level readiness. Consider these scenarios:

• Local Storage Provisioning: A DaemonSet formats and mounts a local SSD. A pod requiring that fast storage must not schedule until the mount is confirmed writable.
• Kernel Module Dependencies: Your workload needs a specific kernel module (e.g., rdma, nf_conntrack, a custom FPGA driver) loaded and configured. The node is healthy, but the module isn’t present.
• Security & Compliance: Your security policy mandates that a host-based intrusion detection system (HIDS) or a data loss prevention (DLP) agent is running and heartbeating healthily on every node before any workload can run.
• Hardware Attestation: In confidential computing or regulated environments, you may need cryptographic proof from a Trusted Platform Module (TPM) or AMD SEV-SNP that the node’s firmware and boot chain are in a known, trusted state.
• External Resource Availability: The node needs access to a licensed software server, a specific network filesystem, or an external hardware security module (HSM).

Scheduling pods without these conditions met leads to runtime errors, inconsistent performance, or policy violations. The Node Readiness Controller, by adding custom conditions to the node’s status, allows you to define these gates explicitly. The Kubernetes scheduler will then treat a node as unschedulable until all conditions—both built-in and custom—are met.

Architecting Custom Readiness Checks

The pattern involves three key components:

1. The Condition: A custom condition name (e.g., node.alexandre-vazquez.com/LocalStorageReady) you define and add to the node’s .status.conditions.
2. The Checker: A controller or agent running on the node (often as a DaemonSet) that performs the actual validation (e.g., tests a mount, checks a kernel module, calls a health API).
3. The Enforcer: The Node Readiness Controller itself, which watches for these conditions and applies a matching taint (e.g., node.alexandre-vazquez.com/local-storage-not-ready:NoSchedule) when the condition is False or Unknown.

Pods that require the specialized resource must then have a corresponding toleration for that taint. This creates a clean, declarative contract: the pod declares its dependency, and the platform guarantees the node meets the requirement before scheduling.

Practical Implementation: Node-Local Storage Readiness

Let’s implement a common pattern: ensuring a local NVMe volume is formatted, mounted, and performance-tested before allowing pods to claim it via a PersistentVolumeClaim.

First, we define a DaemonSet that runs an init container to prepare the storage and a main container that acts as the readiness checker. The checker will periodically validate the mount and update the node’s condition.

1. DaemonSet for Storage Provisioning and Health Checking

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: local-storage-readiness
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: local-storage-readiness
  template:
    metadata:
      labels:
        name: local-storage-readiness
    spec:
      hostPID: true
      containers:
      - name: checker
        image: alpine:latest
        command:
        - "/bin/sh"
        args:
        - "-c"
        - |
          # Function to update node condition
          update_condition() {
            local condition=$1
            local status=$2
            local message=$3
            local patch=$(cat <<EOF
            {
              "status": {
                "conditions": [
                  {
                    "type": "$condition",
                    "status": "$status",
                    "lastHeartbeatTime": "$(date -u +'%Y-%m-%dT%H:%M:%SZ')",
                    "reason": "LocalStorageCheck",
                    "message": "$message"
                  }
                ]
              }
            }
            EOF
            )
            curl -k -X PATCH -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" -H "Content-Type: application/strategic-merge-patch+json" 
              "https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/api/v1/nodes/$(cat /etc/hostname)/status" -d "$patch"
          }

          # Main check loop
          while true; do
            if mountpoint -q /mnt/local-ssd && [ -w /mnt/local-ssd ] && dd if=/dev/zero of=/mnt/local-ssd/test.bin bs=1M count=100 oflag=direct 2>&1 | grep -q 'records out'; then
              update_condition "LocalStorageReady" "True" "/mnt/local-ssd is writable and performant"
            else
              update_condition "LocalStorageReady" "False" "Local storage failed health check"
            fi
            sleep 30
          done
        securityContext:
          privileged: true
        volumeMounts:
        - name: local-ssd
          mountPath: /mnt/local-ssd
        - name: kube-api-access
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
      initContainers:
      - name: provisioner
        image: alpine:latest
        command: ["sh", "-c"]
        args:
          - mkfs.ext4 -F /dev/nvme0n1 && mkdir -p /mnt/local-ssd && mount /dev/nvme0n1 /mnt/local-ssd
        securityContext:
          privileged: true
        volumeMounts:
        - name: local-ssd
          mountPath: /mnt/local-ssd
        - name: device
          mountPath: /dev/nvme0n1
      volumes:
      - name: local-ssd
        hostPath:
          path: /mnt/local-ssd
          type: DirectoryOrCreate
      - name: device
        hostPath:
          path: /dev/nvme0n1
      - name: kube-api-access
        projected:
          sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token

2. NodeReadinessController Configuration

Now, configure the Node Readiness Controller to watch for our LocalStorageReady condition and apply a taint when it’s not True.

apiVersion: v1
kind: ConfigMap
metadata:
  name: node-readiness-controller-config
  namespace: kube-system
data:
  config.yaml: |
    conditions:
    - name: "LocalStorageReady"
      taint:
        key: "node.alexandre-vazquez.com/local-storage-not-ready"
        effect: "NoSchedule"
      state:
        true: "Ready"  # No taint when condition is True
        false: "NotReady"  # Apply taint when condition is False
        unknown: "NotReady" # Apply taint when condition is Unknown

3. Pod Specification with Toleration

A pod that requires this local storage must tolerate the taint, creating an explicit dependency.

apiVersion: v1
kind: Pod
metadata:
  name: data-processor
spec:
  tolerations:
  - key: "node.alexandre-vazquez.com/local-storage-not-ready"
    operator: "Exists"
    effect: "NoSchedule"
  containers:
  - name: app
    image: my-data-app:latest
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    hostPath:
      path: /mnt/local-ssd/pod-data
      type: DirectoryOrCreate

This pattern ensures the pod only schedules onto nodes where the local storage health check has passed. The checker’s simple performance test (dd) helps catch degraded disks that are mounted but failing.

Integrating with Node Feature Discovery and CI/CD

For checks based on hardware capabilities, integrating with Node Feature Discovery (NFD) is powerful. Instead of writing custom hardware detection logic, you can use NFD to label nodes with features (e.g., feature.node.kubernetes.io/cpu-cpuid.AVX512BW: "true"). Your readiness checker can then verify that these expected labels are present and correspond to a working feature.

For example, a checker could:

1. Use NFD labels to identify nodes with specific CPU extensions or GPUs.
2. Run a micro-benchmark or diagnostic (e.g., nvidia-smi --query-gpu=health --format=csv).
3. Set a condition like GpuHealthy to True or False based on the result.

This creates a two-stage validation: NFD discovers the hardware, and your readiness controller certifies its operational health.

CI/CD Pipeline Integration

In a GitOps-driven platform, custom readiness conditions should be treated as first-class citizens. Your deployment pipelines can include steps to verify that target nodes are not only Ready but also meet all custom conditions before proceeding:

# Example pipeline step using kubectl
- name: Validate Node Readiness for Workload
  run: |
    REQUIRED_CONDITIONS=(
      "LocalStorageReady"
      "SecurityAgentReady"
      "FirmwareAttested"
    )
    
    for condition in "${REQUIRED_CONDITIONS[@]}"; do
      if ! kubectl get node $TARGET_NODE -o jsonpath="{.status.conditions[?(@.type=='$condition')].status}" | grep -q "True"; then
        echo "Error: Node $TARGET_NODE not ready. Condition $condition not met."
        exit 1
      fi
    done
    echo "All custom readiness conditions met for $TARGET_NODE."

This proactive check in your CD pipeline can prevent deployments from hanging or failing due to unmet node-level dependencies.

Advanced Use Case: Security Agent Health Gate

Consider a security team that mandates the presence and health of a host-based agent (e.g., Falco, a commercial EDR). A readiness checker DaemonSet can query the agent’s local health API or UNIX socket. If the agent is unresponsive or reports a compromised state, the checker sets a SecurityAgentReady condition to False, tainting the node and preventing any new workloads from scheduling. This hardens your security posture by ensuring no pod runs on a node with a broken security control.

Best Practices and Considerations

• Condition Naming: Use a domain prefix (e.g., yourcompany.com/) to avoid collisions with future Kubernetes built-in conditions.
• Checker Resilience: The checker DaemonSet itself must be highly available and lightweight. Ensure it has appropriate resource requests/limits and a restart policy that maintains the check.
• Taint Effects: Use NoSchedule to prevent new pods. For immediate response to a condition breaking on a running node, you might also consider NoExecute with a tolerationSeconds value on pods, but this requires careful design to avoid unnecessary churn.
• Combining Conditions: A node can have multiple custom conditions. The scheduler respects all of them; a pod will only schedule if it tolerates all the taints applied due to unmet conditions.
• Monitoring and Alerting: Treat custom condition states as critical metrics. Alert your platform team when a False condition persists, indicating a node-level issue requiring investigation.

Frequently Asked Questions

Conclusion

The Node Readiness Controller is a gateway to a more robust, intentional scheduling model. By moving beyond the assumption that a Kubernetes Ready node is ready for your workloads, you can enforce the precise prerequisites your applications demand. Whether it’s certified hardware, validated storage, verified security controls, or licensed software, custom readiness conditions allow your platform to make smarter scheduling decisions.

This shifts the responsibility from the application developer (who must write complex initialization and error-handling logic) to the platform (which guarantees a capable environment). The result is fewer runtime surprises, more consistent performance, and a cluster that truly aligns with your operational and compliance requirements. Start by identifying one critical, non-standard dependency in your most important workloads, and implement a readiness gate for it. The pattern will quickly prove its value across your entire infrastructure.