Lecture on Affinity, Anti-Affinity, Taints and Tolerations, and Workload Placement in the Cluster

KUBERNATES COURSE

When understood and designed correctly, workload placement in a Kubernetes cluster can improve reliability, performance, and cost. In this lesson we explore how to guide the scheduler using Affinity, Anti-Affinity, Taints, and Tolerations, and how to combine them to achieve effective distribution, isolation, and colocation.

Why placement matters

The Kubernetes scheduler decides where to run Pods based on resources, constraints, and preferences. Intentional placement helps avoid bottlenecks, reduce latency between related services, isolate sensitive workloads, and prevent single points of failure. Without clear rules, the cluster can look “healthy,” yet services may suffer from noisy neighbors, zone imbalance, and gradual degradation.

Key scheduling concepts

The scheduler evaluates each Pod against candidate nodes, filtering incompatible ones (filtering phase) and then scoring the remaining ones (scoring phase). Some directives are hard constraints (must be satisfied), others are soft preferences (improve the score). Node and Pod labels, together with specific manifest fields, guide these decisions.

Node Affinity and Pod Affinity

Node Affinity

Allows you to require or prefer nodes with certain labels. Use it when you want a Pod to run, for example, only in a specific zone or on nodes with specific hardware (GPU, SSD, dedicated CPU).

Main modes:

  • requiredDuringSchedulingIgnoredDuringExecution: hard constraint at scheduling time; if not satisfied, the Pod does not start.
  • preferredDuringSchedulingIgnoredDuringExecution: soft preference; the scheduler tries to satisfy it, but may place the Pod elsewhere if needed.

Typical example: constrain a stateful service to nodes with local NVMe disks or distribute workloads across zones using the topology.kubernetes.io/zone label.

Pod Affinity

Colocation between Pods is useful when services communicate frequently and you want to minimize latency or network costs. Pod Affinity specifies that a Pod must or should be scheduled close (same node, zone, region) to Pods that match a label selector.

As with the node variant, there are requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution. Pod affinity uses a topologyKey to define the “domain” (e.g., kubernetes.io/hostname for the same node, topology.kubernetes.io/zone for the same zone).

Pod Anti-Affinity

It is the counterpart for separation. It allows you to distribute replicas across different domains. Typically you enforce that two replicas of the same application do not coexist on the same node or in the same zone, reducing the risk of simultaneous downtime.

Common use: with a 3-replica Deployment, set a “required” anti-affinity on the same hostname to force spreading across nodes. With zone domains, you gain resilience to zone failures.

Taints and Tolerations

Taints mark the node to repel non-tolerating Pods. Tolerations, defined on the Pod, allow landing on tainted nodes. It is a push mechanism from the nodes’ perspective (isolate the node), complementary to affinity/anti-affinity which is pull from the Pods’ perspective (choose the nodes).

Typical taint effects:

  • NoSchedule: Pods without a toleration are not scheduled onto that node.
  • PreferNoSchedule: placement is avoided if possible.
  • NoExecute: in addition to blocking new Pods, it evicts existing non-tolerating ones; with tolerationSeconds you can delay eviction.

This tool is ideal for reserving nodes for specific workloads (e.g., batch, GPU, isolated environments) or for draining/segregating problematic nodes.

When to use what

  • Strong isolation: apply a taint on the node and add a toleration only to authorized Pods.
  • Pod colocation/separation: use (anti-)affinity with the appropriate topologyKey.
  • Hardware or topology constraints: use node affinity on stable labels (zone, instance type, accelerators).
  • Balancing preferences: use the “preferred” variants to avoid starvation.

Practical examples

Node Affinity + Pod Anti-Affinity

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values: ["eu-west-1a", "eu-west-1b"]
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 50
            preference:
              matchExpressions:
              - key: disktype
                operator: In
                values: ["nvme"]
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: web
            topologyKey: kubernetes.io/hostname
      containers:
      - name: app
        image: nginx:1.25
  

Taints and Tolerations

We apply a taint to dedicate nodes to databases and tolerate the taint only in DB Pods.

# Node taint
kubectl taint nodes node-gpu-1 dedicated=db:NoSchedule

# Pod that can land on that node
apiVersion: v1
kind: Pod
metadata:
  name: postgres
  labels:
    tier: db
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "db"
    effect: "NoSchedule"
  containers:
  - name: postgres
    image: postgres:16
  

Topology-based distribution (spread)

To evenly distribute replicas across zones or nodes, you can use Topology Spread Constraints. They complement anti-affinity and are often more efficient.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 6
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: api
      containers:
      - name: api
        image: ghcr.io/example/api:1.0.0
  

Best practices

  • Use clear, stable labels: topology.kubernetes.io/zone, kubernetes.io/hostname, accelerators such as cloud.google.com/gke-accelerator or nvidia.com/gpu.
  • Prefer “preferredDuringScheduling…” when possible: it reduces the risk that Pods remain Pending.
  • Combine spread constraints and anti-affinity in moderation: too many constraints can drastically reduce effective capacity.
  • Avoid hardcoded nodeName: it is brittle and bypasses the scheduler.
  • Version and document policies: share reusable patterns across teams.
  • Monitor scheduling events: Pending errors, causes of Insufficient CPU/MEM, unmet constraints.
  • Balance resilience and costs: spreading across zones can increase latency and the price of inter-zone traffic.

Common mistakes to avoid

  • “Required” anti-affinity with an overly broad topologyKey on large clusters: can degrade scheduling performance.
  • Taints applied without a tolerations plan: you risk creating bottlenecks and Pods stuck in Pending.
  • Unstable labels on nodes: if they change dynamically, Pods may not be replaced consistently.
  • No fallback: only hard constraints without preferences reduce flexibility during spikes or maintenance.

Observability and troubleshooting

To analyze why a Pod is not being scheduled, use kubectl describe pod and check the Events section for messages about unmet constraints. Inspect node labels with kubectl get nodes -L … to confirm the presence of required labels. Cluster monitoring tools (Scheduler metrics, Events, dashboards) help detect zone skew, hot spots, and prolonged Pending.

Putting it all together

A robust design typically combines: node affinity for hardware/topology constraints, pod anti-affinity or topology spread for resilience, taints to isolate classes of workloads, and soft preferences to maintain flexibility. Iterate with small changes, validate in staging environments, and measure the impact on latency, throughput, and costs.

Call to action

Ready to optimize scheduling in your cluster? Map node labels, identify services that require colocation or isolation, and apply the patterns shown. If you want a review of your policies or a training session dedicated to your team, contact us: together we will build an effective, resilient, and sustainable placement strategy.

Se vuoi farmi qualche richiesta o contattarmi per un aiuto riempi il seguente form

    Comments