5.3. Scheduling

In this lab, you decide on which nodes your Pods run: you pin them to a node, spread them across all nodes and keep them off a node with a taint.

Step 1: Pin Pods to a Node

Create the directory of this lab and change into it:

mkdir -p ~/labs/5.3
cd ~/labs/5.3

Label the node k3d-labs-agent-1 as a node with a fast disk:

kubectl label node k3d-labs-agent-1 disk=ssd
node/k3d-labs-agent-1 labeled
kubectl get nodes -L disk
NAME                STATUS   ROLES                AGE     VERSION        DISK
k3d-labs-agent-0    Ready    <none>               7h20m   v1.35.5+k3s1
k3d-labs-agent-1    Ready    <none>               7h20m   v1.35.5+k3s1   ssd
k3d-labs-server-0   Ready    control-plane,etcd   7h20m   v1.35.5+k3s1

Create the file pinned.yaml in ~/labs/5.3 with a Deployment whose Pods only run on nodes with this label:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pinned
spec:
  replicas: 2
  selector:
    matchLabels:
      app: pinned
  template:
    metadata:
      labels:
        app: pinned
    spec:
      nodeSelector:
        disk: ssd
      containers:
        - name: nginx
          image: nginx:1.29

Apply it and list the Pods with their nodes:

kubectl apply -f pinned.yaml
deployment.apps/pinned created
kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE               NOMINATED NODE   READINESS GATES
pinned-d6ff6fd69-bf9zn   1/1     Running   0          1s    10.42.1.181   k3d-labs-agent-1   <none>           <none>
pinned-d6ff6fd69-ssn84   1/1     Running   0          1s    10.42.1.180   k3d-labs-agent-1   <none>           <none>

If the Pods aren’t Running yet, wait a moment and repeat the command.

Explanation

kubectl label node adds a label to a node, just like to a Pod, and -L disk shows its value as a column. nodeSelector limits the Pods to nodes that carry all the listed labels, so the scheduler had only one node to choose from, and both Pods landed on k3d-labs-agent-1. That’s useful for hardware that only some nodes have, but bad for availability: if this node fails, all Pods are gone.


Step 2: Spread the Replicas Across Nodes

Create the file spread.yaml in ~/labs/5.3 with a Deployment that spreads its Pods evenly over the nodes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spread
spec:
  replicas: 3
  selector:
    matchLabels:
      app: spread
  template:
    metadata:
      labels:
        app: spread
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: spread
      containers:
        - name: nginx
          image: nginx:1.29

Apply it and list its Pods with their nodes:

kubectl apply -f spread.yaml
deployment.apps/spread created
kubectl get pods -o wide -l app=spread
NAME                      READY   STATUS    RESTARTS   AGE   IP            NODE                NOMINATED NODE   READINESS GATES
spread-7ffbd6ffdc-nx9q5   1/1     Running   0          2s    10.42.2.69    k3d-labs-server-0   <none>           <none>
spread-7ffbd6ffdc-w2v82   1/1     Running   0          2s    10.42.1.182   k3d-labs-agent-1    <none>           <none>
spread-7ffbd6ffdc-zpzjp   1/1     Running   0          2s    10.42.0.131   k3d-labs-agent-0    <none>           <none>

If the Pods aren’t Running yet, wait a moment and repeat the command.

Explanation

Each node runs one Pod. The rule counts the Pods with the label app: spread per node:

FieldMeaning
topologyKey: kubernetes.io/hostnameSpread per node: every node carries this label with its own name
maxSkew: 1The number of Pods on two nodes may differ by at most one
whenUnsatisfiable: DoNotScheduleIf no node fits the rule, the Pod stays Pending instead of breaking it

If one node fails, two of the three Pods keep running. Without the rule, the scheduler also tries to spread the Pods, but other criteria such as free memory can win.


Step 3: Keep Pods off a Node With a Taint

Taint the node k3d-labs-agent-1, so that it only accepts Pods that tolerate the taint:

kubectl taint node k3d-labs-agent-1 dedicated=ssd:NoSchedule
node/k3d-labs-agent-1 tainted

Scale the Deployment pinned to three replicas and list its Pods:

kubectl scale deployment pinned --replicas=3
deployment.apps/pinned scaled
kubectl get pods -o wide -l app=pinned
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE               NOMINATED NODE   READINESS GATES
pinned-d6ff6fd69-bf9zn   1/1     Running   0          8s    10.42.1.181   k3d-labs-agent-1   <none>           <none>
pinned-d6ff6fd69-g7vfr   0/1     Pending   0          5s    <none>        <none>             <none>           <none>
pinned-d6ff6fd69-ssn84   1/1     Running   0          8s    10.42.1.180   k3d-labs-agent-1   <none>           <none>

Find out why the new Pod is Pending. Use the name of your Pending Pod:

kubectl describe pod pinned-d6ff6fd69-g7vfr
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  5s    default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Let the Pods of pinned tolerate the taint: in pinned.yaml, set replicas to 3 and add tolerations below the nodeSelector:

spec:
  replicas: 3
      nodeSelector:
        disk: ssd
      tolerations:
        - key: dedicated
          operator: Equal
          value: ssd
          effect: NoSchedule

Apply the file and list the Pods again:

kubectl apply -f pinned.yaml
deployment.apps/pinned configured
kubectl get pods -o wide -l app=pinned
NAME                      READY   STATUS    RESTARTS   AGE   IP            NODE               NOMINATED NODE   READINESS GATES
pinned-75df689645-hrwtq   1/1     Running   0          6s    10.42.1.183   k3d-labs-agent-1   <none>           <none>
pinned-75df689645-t48dg   1/1     Running   0          5s    10.42.1.184   k3d-labs-agent-1   <none>           <none>
pinned-75df689645-xkgtg   1/1     Running   0          4s    10.42.1.185   k3d-labs-agent-1   <none>           <none>

If not all three Pods are Running yet, wait a moment and repeat the command.

Explanation

A taint is the opposite of a label in a nodeSelector: it keeps away every Pod that doesn’t explicitly tolerate it. dedicated=ssd is the key and value of the taint, NoSchedule its effect. The new Pod had nowhere to go: its nodeSelector allowed only k3d-labs-agent-1, and the taint kept it off there. The scheduler’s message lists the reason for each node. The part about preemption says that evicting other Pods wouldn’t help either.

NoSchedule affects only new Pods: the two Pods that ran before the taint, and the Pod of spread on the same node, kept running. The toleration lets the Pods of pinned in again. It changed the Pod template, so the Deployment replaced all three Pods, as with the new image in 2.4. Together, the taint and the toleration reserve the node: only Pods that tolerate the taint may run there, and the nodeSelector makes sure that they do.

kubectl scale changed the number of replicas only in the cluster, not in your file. That’s why you set replicas: 3 in the file, too: otherwise kubectl apply would have scaled the Deployment back to 2.


Optional Tasks

Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need. Open the solution if you get stuck.

Task 1: Built-in Labels

Every node already carries a few labels. Find the one that marks the control plane node and run a Deployment control-plane with two replicas only there.

CommandEffect
kubectl get node <name> --show-labelsShows all labels of the node
Solution

Show the labels of the server node:

kubectl get node k3d-labs-server-0 --show-labels
NAME                STATUS   ROLES                AGE     VERSION        LABELS
k3d-labs-server-0   Ready    control-plane,etcd   7h22m   v1.35.5+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3d-labs-server-0,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/etcd=true,node.kubernetes.io/instance-type=k3s

Create the file control-plane.yaml in ~/labs/5.3 and select the label node-role.kubernetes.io/control-plane:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: control-plane
spec:
  replicas: 2
  selector:
    matchLabels:
      app: control-plane
  template:
    metadata:
      labels:
        app: control-plane
    spec:
      nodeSelector:
        node-role.kubernetes.io/control-plane: "true"
      containers:
        - name: nginx
          image: nginx:1.29
kubectl apply -f ~/labs/5.3/control-plane.yaml
deployment.apps/control-plane created
kubectl get pods -o wide -l app=control-plane
NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
control-plane-56957b9fd4-lhxgw   1/1     Running   0          1s    10.42.2.78   k3d-labs-server-0   <none>           <none>
control-plane-56957b9fd4-n4fqh   1/1     Running   0          1s    10.42.2.77   k3d-labs-server-0   <none>           <none>

Kubernetes sets labels for the operating system, the architecture and the host name. kubernetes.io/hostname is the label that the spread rule in Step 2 counts by. The quotes around "true" matter: label values are strings.

Task 2: Soft Preference

nodeSelector is a hard rule. Create a Deployment preferred with two replicas that only prefers nodes with the label disk: ssd. Where do its Pods end up while k3d-labs-agent-1 is tainted?

The rule is a node affinity in the Pod spec:

      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: disk
                    operator: In
                    values:
                      - ssd

You know all the commands you need.

Solution

Create the file preferred.yaml in ~/labs/5.3:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: preferred
spec:
  replicas: 2
  selector:
    matchLabels:
      app: preferred
  template:
    metadata:
      labels:
        app: preferred
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: disk
                    operator: In
                    values:
                      - ssd
      containers:
        - name: nginx
          image: nginx:1.29
kubectl apply -f ~/labs/5.3/preferred.yaml
deployment.apps/preferred created
kubectl get pods -o wide -l app=preferred
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE                NOMINATED NODE   READINESS GATES
preferred-bff857467-vlmnx   1/1     Running   0          1s    10.42.0.137   k3d-labs-agent-0    <none>           <none>
preferred-bff857467-x4s26   1/1     Running   0          1s    10.42.2.75    k3d-labs-server-0   <none>           <none>

The preferred node is tainted, so the scheduler picked other nodes instead of leaving the Pods Pending: a preference only adds points when the scheduler ranks the nodes. With requiredDuringScheduling..., node affinity becomes a hard rule like nodeSelector, but with more operators, e.g. In with several values or NotIn.

Task 3: One per Node

Create a Deployment one-per-node with three replicas that never runs two of its Pods on the same node. How many of them run while k3d-labs-agent-1 is tainted?

The rule is a pod anti-affinity in the Pod spec:

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app: one-per-node

You know all the commands you need.

Solution

Create the file one-per-node.yaml in ~/labs/5.3:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: one-per-node
spec:
  replicas: 3
  selector:
    matchLabels:
      app: one-per-node
  template:
    metadata:
      labels:
        app: one-per-node
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app: one-per-node
      containers:
        - name: nginx
          image: nginx:1.29
kubectl apply -f ~/labs/5.3/one-per-node.yaml
deployment.apps/one-per-node created
kubectl get pods -o wide -l app=one-per-node
NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE                NOMINATED NODE   READINESS GATES
one-per-node-78bb644999-jvcxs   1/1     Running   0          10s   10.42.0.138   k3d-labs-agent-0    <none>           <none>
one-per-node-78bb644999-n6jpv   1/1     Running   0          10s   10.42.2.76    k3d-labs-server-0   <none>           <none>
one-per-node-78bb644999-x494z   0/1     Pending   0          10s   <none>        <none>              <none>           <none>

kubectl describe on the Pending Pod shows the reason:

  Warning  FailedScheduling  10s   default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match pod anti-affinity rules. ...

Only two nodes accept the Pods, and the anti-affinity forbids a second Pod of one-per-node on either of them. A spread rule with maxSkew: 1 would allow two Pods on one node as soon as all nodes have one. The anti-affinity never does.


Clean Up

Delete the Deployments:

kubectl delete -f ~/labs/5.3/
deployment.apps "pinned" deleted from default namespace
deployment.apps "spread" deleted from default namespace

If you did the optional tasks, the output also lists control-plane, preferred and one-per-node. Remove the taint and the label (the - at the end removes them):

kubectl taint node k3d-labs-agent-1 dedicated=ssd:NoSchedule-
node/k3d-labs-agent-1 untainted
kubectl label node k3d-labs-agent-1 disk-
node/k3d-labs-agent-1 unlabeled

Complete files

~/labs/5.3/pinned.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pinned
spec:
  replicas: 3
  selector:
    matchLabels:
      app: pinned
  template:
    metadata:
      labels:
        app: pinned
    spec:
      nodeSelector:
        disk: ssd
      tolerations:
        - key: dedicated
          operator: Equal
          value: ssd
          effect: NoSchedule
      containers:
        - name: nginx
          image: nginx:1.29

~/labs/5.3/spread.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spread
spec:
  replicas: 3
  selector:
    matchLabels:
      app: spread
  template:
    metadata:
      labels:
        app: spread
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: spread
      containers:
        - name: nginx
          image: nginx:1.29