In this lab, you decide on which nodes your Pods run: you pin them to a node, spread them across all nodes and keep
them off a node with a taint.
Step 1: Pin Pods to a Node
Create the directory of this lab and change into it:
mkdir -p ~/labs/5.3
cd ~/labs/5.3
Label the node k3d-labs-agent-1 as a node with a fast disk:
kubectl label node k3d-labs-agent-1 disk=ssd
node/k3d-labs-agent-1 labeled
kubectl get nodes -L disk
NAME STATUS ROLES AGE VERSION DISK
k3d-labs-agent-0 Ready <none> 7h20m v1.35.5+k3s1
k3d-labs-agent-1 Ready <none> 7h20m v1.35.5+k3s1 ssd
k3d-labs-server-0 Ready control-plane,etcd 7h20m v1.35.5+k3s1
Create the file pinned.yaml in ~/labs/5.3 with a Deployment whose Pods only run on nodes with this
label:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pinned-d6ff6fd69-bf9zn 1/1 Running 0 1s 10.42.1.181 k3d-labs-agent-1 <none> <none>
pinned-d6ff6fd69-ssn84 1/1 Running 0 1s 10.42.1.180 k3d-labs-agent-1 <none> <none>
If the Pods aren’t Running yet, wait a moment and repeat the command.
Explanation
kubectl label node adds a label to a node, just like to a Pod, and -L disk shows its value as a column.
nodeSelector limits the Pods to nodes that carry all the listed labels, so the scheduler had only one node to
choose from, and both Pods landed on k3d-labs-agent-1. That’s useful for hardware that only some nodes have, but
bad for availability: if this node fails, all Pods are gone.
Step 2: Spread the Replicas Across Nodes
Create the file spread.yaml in ~/labs/5.3 with a Deployment that spreads its Pods evenly over the
nodes:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
spread-7ffbd6ffdc-nx9q5 1/1 Running 0 2s 10.42.2.69 k3d-labs-server-0 <none> <none>
spread-7ffbd6ffdc-w2v82 1/1 Running 0 2s 10.42.1.182 k3d-labs-agent-1 <none> <none>
spread-7ffbd6ffdc-zpzjp 1/1 Running 0 2s 10.42.0.131 k3d-labs-agent-0 <none> <none>
If the Pods aren’t Running yet, wait a moment and repeat the command.
Explanation
Each node runs one Pod. The rule counts the Pods with the label app: spread per node:
Field
Meaning
topologyKey: kubernetes.io/hostname
Spread per node: every node carries this label with its own name
maxSkew: 1
The number of Pods on two nodes may differ by at most one
whenUnsatisfiable: DoNotSchedule
If no node fits the rule, the Pod stays Pending instead of breaking it
If one node fails, two of the three Pods keep running. Without the rule, the scheduler also tries to spread the Pods,
but other criteria such as free memory can win.
Step 3: Keep Pods off a Node With a Taint
Taint the node k3d-labs-agent-1, so that it only accepts Pods that tolerate the taint:
Scale the Deployment pinned to three replicas and list its Pods:
kubectl scale deployment pinned --replicas=3
deployment.apps/pinned scaled
kubectl get pods -o wide -l app=pinned
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pinned-d6ff6fd69-bf9zn 1/1 Running 0 8s 10.42.1.181 k3d-labs-agent-1 <none> <none>
pinned-d6ff6fd69-g7vfr 0/1 Pending 0 5s <none> <none> <none> <none>
pinned-d6ff6fd69-ssn84 1/1 Running 0 8s 10.42.1.180 k3d-labs-agent-1 <none> <none>
Find out why the new Pod is Pending. Use the name of your Pending Pod:
kubectl describe pod pinned-d6ff6fd69-g7vfr
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5s default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Let the Pods of pinned tolerate the taint: in pinned.yaml, set replicas to 3 and add
tolerations below the nodeSelector:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pinned-75df689645-hrwtq 1/1 Running 0 6s 10.42.1.183 k3d-labs-agent-1 <none> <none>
pinned-75df689645-t48dg 1/1 Running 0 5s 10.42.1.184 k3d-labs-agent-1 <none> <none>
pinned-75df689645-xkgtg 1/1 Running 0 4s 10.42.1.185 k3d-labs-agent-1 <none> <none>
If not all three Pods are Running yet, wait a moment and repeat the command.
Explanation
A taint is the opposite of a label in a nodeSelector: it keeps away every Pod that doesn’t explicitly tolerate it.
dedicated=ssd is the key and value of the taint, NoSchedule its effect.
The new Pod had nowhere to go: its nodeSelector allowed only k3d-labs-agent-1, and the taint kept it off there.
The scheduler’s message lists the reason for each node. The part about preemption says that evicting other Pods
wouldn’t help either.
NoSchedule affects only new Pods: the two Pods that ran before the taint, and the Pod of spread on the same node,
kept running. The toleration lets the Pods of pinned in again. It changed the Pod template, so the Deployment
replaced all three Pods, as with the new image in 2.4. Together, the taint and the toleration reserve the node: only
Pods that tolerate the taint may run there, and the nodeSelector makes sure that they do.
kubectl scale changed the number of replicas only in the cluster, not in your file. That’s why you set
replicas: 3 in the file, too: otherwise kubectl apply would have scaled the Deployment back to 2.
Optional Tasks
Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need.
Open the solution if you get stuck.
Task 1: Built-in Labels
Every node already carries a few labels. Find the one that marks the control plane node and run a Deployment
control-plane with two replicas only there.
Command
Effect
kubectl get node <name> --show-labels
Shows all labels of the node
Solution
Show the labels of the server node:
kubectl get node k3d-labs-server-0 --show-labels
NAME STATUS ROLES AGE VERSION LABELS
k3d-labs-server-0 Ready control-plane,etcd 7h22m v1.35.5+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3d-labs-server-0,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/etcd=true,node.kubernetes.io/instance-type=k3s
Create the file control-plane.yaml in ~/labs/5.3 and select the label
node-role.kubernetes.io/control-plane:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
control-plane-56957b9fd4-lhxgw 1/1 Running 0 1s 10.42.2.78 k3d-labs-server-0 <none> <none>
control-plane-56957b9fd4-n4fqh 1/1 Running 0 1s 10.42.2.77 k3d-labs-server-0 <none> <none>
Kubernetes sets labels for the operating system, the architecture and the host name. kubernetes.io/hostname is
the label that the spread rule in Step 2 counts by. The quotes around "true" matter: label values are strings.
Task 2: Soft Preference
nodeSelector is a hard rule. Create a Deployment preferred with two replicas that only prefers nodes with the
label disk: ssd. Where do its Pods end up while k3d-labs-agent-1 is tainted?
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
preferred-bff857467-vlmnx 1/1 Running 0 1s 10.42.0.137 k3d-labs-agent-0 <none> <none>
preferred-bff857467-x4s26 1/1 Running 0 1s 10.42.2.75 k3d-labs-server-0 <none> <none>
The preferred node is tainted, so the scheduler picked other nodes instead of leaving the Pods Pending: a
preference only adds points when the scheduler ranks the nodes. With requiredDuringScheduling..., node affinity
becomes a hard rule like nodeSelector, but with more operators, e.g. In with several values or NotIn.
Task 3: One per Node
Create a Deployment one-per-node with three replicas that never runs two of its Pods on the same node. How many of
them run while k3d-labs-agent-1 is tainted?
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
one-per-node-78bb644999-jvcxs 1/1 Running 0 10s 10.42.0.138 k3d-labs-agent-0 <none> <none>
one-per-node-78bb644999-n6jpv 1/1 Running 0 10s 10.42.2.76 k3d-labs-server-0 <none> <none>
one-per-node-78bb644999-x494z 0/1 Pending 0 10s <none> <none> <none> <none>
kubectl describe on the Pending Pod shows the reason:
Warning FailedScheduling 10s default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match pod anti-affinity rules. ...
Only two nodes accept the Pods, and the anti-affinity forbids a second Pod of one-per-node on either of them. A
spread rule with maxSkew: 1 would allow two Pods on one node as soon as all nodes have one. The anti-affinity
never does.
Clean Up
Delete the Deployments:
kubectl delete -f ~/labs/5.3/
deployment.apps "pinned" deleted from default namespace
deployment.apps "spread" deleted from default namespace
If you did the optional tasks, the output also lists control-plane, preferred and one-per-node. Remove the
taint and the label (the - at the end removes them):