What is Kubernetes Node Affinity? Benefits and Core Concepts
Kubernetes node affinity is an essential scheduling feature that allows you to control pod placement based on node labels and properties. By using node affinity rules, you can specify constraints on which nodes pods can be scheduled, enabling you to optimize resource allocation and enhance performance.
Node affinity works by allowing you to define rules for pod scheduling based on node labels. When defining node affinity rules, you have two options: required and preferred rules. Required rules ensure that pods are scheduled only on nodes that satisfy the defined criteria. If no suitable node is available, the pod remains unscheduled. On the other hand, preferred rules provide a soft constraint and attempt to schedule pods on nodes that match the specified criteria. However, if no such node is available, the pod can still be scheduled on other nodes.
Node affinity rules are an “expanded” option of the simply way by using node selectors. Node selectors are a simple form of node affinity that allows you to assign labels to nodes and match those labels with selectors defined in the pod specification. By specifying a node selector, you can ensure that pods are scheduled only on nodes with matching labels. Node selectors are useful for basic affinity requirements but lack the flexibility and fine-grained control provided by more advanced affinity options.
Node Affinity Trade-offs: Required vs Preferred Rules and Failure Scenarios
But this awesome capability has some trade-offs that you need to take in consideration because nothing comes with a price that you need to be aware of, so, let’s go to the important question, what is the worst case scenario of using any of those options?
Consider a stateful workload, like a distributed database (e.g., etcd or ZooKeeper), deployed with three replicas for consensus and fault tolerance. So you decide to define a set of nodes for this workload and use node affinity rules to ensure the pods are scheduled to those nodes. And, you need to think: should I use the preferred mode or the requiredMode?
Let’s say that you go with the required option and you define it like this, what happen if one of your nodes goes down? The pod will be try to be rescheduled again and unless there are another node “with same label” to that, it cannot be deployed? If you additional defined a pod anti-affinity rule to ensure each of the replicas is in a different host to ensure that in case that one node is going down you lose only a single replica, you’re losing the option to rescheudle the workload even if you have another nodes without the label available. So, you’re not in a so reliable option.
Ok, so you go with the preferred to ensure that you workload is for sure scheduled even if it is in another node, and in that case you can end up on the situation that those nodes are scheduled on other nodes keeping those nodes with the proper label without the workload that they should have, making the situation strange and more difficult to administer because you cannot ensure your workloads is on the nodes that you expected to be.
Additional to that, if the nodes has even taints to ensure other workloads cannot be placed there, you can end up in a situation that the “labeled-pods” are scheduled on non-labeled nodes, and the non-labeled pods cannot use the nodes because they’re tainted and can be not be able to use the un-labeled ones if there are not enough resources. So you’re generating an impact on the other workloasd and potentially affecting the schedulling of the other workloads.
Preparing for Unexpected Outages with Node Affinity
So, as you can see, each decision has some disadvatanges that you need to take in consdieration before defining those rules, because if you don’t, you will figure it out when this happen on an production enviornment probably as a result of some unexpected outage, because we all know that in the meantime that nothing bad happens everything works as expected, but the potential of these solutions and its reason to be used is exactly to provide the tools and the options to be prepared when bad things happens.
So, next time that you need to define a node affinity rule try to think about the disadvantages of each of the option and try to select that one that works best for you and mitigate the problems that it can bring to the table of your production environment.
📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.
Frequently Asked Questions
What is the difference between nodeSelector and node affinity in Kubernetes?
nodeSelector is a simple field that requires a node to have all specified labels. Node affinity is a more expressive API that supports complex operators like In, NotIn, and Exists, and distinguishes between hard (requiredDuringScheduling...) and soft (preferredDuringScheduling...) constraints. Use nodeSelector for basic needs; use node affinity for advanced scheduling logic.
When should I use required vs preferred node affinity rules?
Use required rules for strict placement needs, like licensing constraints or specific hardware (e.g., GPU nodes). Use preferred rules for optimization, like trying to place pods on nodes in the same availability zone for lower latency. Be aware that required rules can prevent scheduling during node failures, while preferred rules may not guarantee optimal placement.
What are the risks of using required node affinity?
The primary risk is scheduling failure. If no node matches the required rules (e.g., due to a failure or label mismatch), the pod will remain Pending. This can lead to application downtime, especially if combined with Pod Anti-Affinity, which further restricts eligible nodes. Always ensure you have enough labeled nodes to handle failures.
How does node affinity interact with taints and tolerations?
They work sequentially. First, the scheduler filters nodes based on node affinity/selector rules. Then, from the filtered nodes, it checks taints and tolerations. A pod will only be scheduled on a node that satisfies both its affinity/selector requirements and for which the pod has a matching toleration for all the node’s taints.
What are best practices for defining node affinity labels?
Use clear, descriptive label keys (e.g., node.kubernetes.io/instance-type, topology.kubernetes.io/zone). Prefer built-in labels where possible. Document the purpose of custom labels. Combine node affinity with pod anti-affinity carefully to avoid over-constraining the scheduler. Test scenarios with node failures.