Node Selectors and Node Affinity: Telling the Scheduler Where Pods Can Run

Node selectors and node affinity are two Kubernetes features that help specify where a pod can run based on node labels. Node selectors use a simple key-value matching approach, while node affinity offers more advanced rules such as IN, NotIn, Exists, DoesNotExist, Gt, and Lt.

Node affinity can be applied in two ways: requiredDuringSchedulingIgnoredDuringExecution (a hard rule, similar to node selectors) and preferredDuringSchedulingIgnoredDuringExecution (a soft rule that nudges the scheduler's scoring without blocking the pod). The latter allows you to assign a weight (1-100) to influence the scheduler's decision.

One pitfall to be aware of is the distinction between matchExpressions and matchFields. matchExpressions work with node labels, while matchFields operate on node fields, like the node's name. It's generally recommended to use matchExpressions and labels rather than matchFields, as the latter is typically used for pinning a pod to a specific node by name.

Node selection is relevant in scenarios where certain nodes have specific capabilities, such as GPU-equipped nodes for GPU-intensive workloads, SSD-backed storage for databases or caches, or specific zones for compliance or licensing reasons. In such cases, using node selectors or node affinity is essential to ensure proper scheduling of pods.

Both node selectors and node affinity are evaluated during the filtering phase of the scheduler. Nodes that don't match the specified labels are removed before the scoring phase begins. The affinity rule with the preferredDuringSchedulingIgnoredDuringExecution setting does not participate in the filtering phase; instead, it adds its weight to a node's score if the node matches.

An important characteristic of both mechanisms is that they are IgnoredDuringExecution. This means that changes to node labels or fields after a pod has been scheduled do not affect the already running pod.

Node selectors are simpler and easier to read, making them suitable when the requirement is straightforward, i.e., a pod must have a specific label. On the other hand, node affinity provides more flexibility for complex scenarios involving OR logic, negation, or preferential scheduling. However, it's essential to remember that pod-to-pod placement and workload distribution are not addressed by these features. Those are covered by pod affinity and anti-affinity, which will be discussed in a later part of this series.

A common mistake when dealing with node selectors and node affinity is setting a required rule without any matching nodes, causing the pod to remain in a Pending state indefinitely. In such cases, reviewing the Events section of the pod using kubectl describe pod can help identify the scheduling failure reason. Another lesser-known issue is that changes to node labels or fields after a pod has been scheduled do not impact the running pod, as these mechanisms are IgnoredDuringExecution.