2.9. Health and Resources
In this lab, you tell Kubernetes when your application is ready and alive, how much CPU and memory it needs, and let it scale the application automatically under load.
Step 1: Check Readiness with a Probe
Create a directory for the files of this lab and change into it:
Create the file web.yaml in ~/labs/2.9. The whoami application from 2.6 answers on /health
and lets you switch that answer to an error, so you can make a Pod unhealthy on purpose:
Create the file web-service.yaml with a Service for the Pods:
Create the Deployment:
Create the Service:
List the Pods:
If the Pods aren’t 1/1 yet, wait a moment and repeat the command.
Forward a port to the first Pod (use a name from your output). The command keeps running:
In a second terminal, switch the health check of this Pod to the error code 500:
curl prints nothing. The first terminal shows Handling connection for 8080. Wait about 15 seconds and list the
Pods again:
If the Pod is still 1/1, wait a few seconds and repeat the command.
Check which Pods the Service sends requests to:
Switch the health check back to 200:
Wait about 10 seconds and list the Pods again:
If the Pod is still 0/1, wait a moment and repeat the command.
Stop the port forwarding with Ctrl+C.
Explanation
The kubelet calls the readinessProbe every 5 seconds (periodSeconds). After three failed calls in a row (the
default failureThreshold), it marks the Pod as not ready: READY 0/1. The Service removes a Pod that isn’t ready
from its endpoints, so only the healthy Pod gets requests now. The unhealthy Pod keeps running and gets requests again
as soon as the probe succeeds.
kubectl port-forward pod/... connects to the Pod directly, not through the Service. That’s why your curl still
reached it.
Step 2: Restart a Broken Container
Add a livenessProbe to the container in web.yaml, right below the readinessProbe:
In the first terminal, apply the manifest:
List the new Pods:
If the old Pods are still listed, wait a moment and repeat the command. Forward a port to one of the new Pods:
In the second terminal, switch its health check to 500 again. The command prints nothing:
Watch the Pods right away:
After about 15 seconds, the liveness probe restarts the container (RESTARTS 1). The readiness probe calls the same
endpoint and fails at about the same time, so the line with 0/1 and RESTARTS 0 may be missing in your output.
Stop the watch and the port forwarding with Ctrl+C.
Explanation
The livenessProbe calls the same endpoint. After three failed calls, the kubelet restarts the container
(RESTARTS 1). The new process of whoami starts healthy, so both probes succeed again.
| Probe | Question | When it fails |
|---|---|---|
readinessProbe | Can the Pod handle requests? | The Service stops sending requests |
livenessProbe | Is the container still alive? | The kubelet restarts the container |
Step 3: Give a Slow Application Time to Start
Create the file slow.yaml with a Pod that needs 20 seconds to start. When its start is complete, it
creates the file /tmp/started, and its liveness probe checks for that file with a command:
In the first terminal, create the Pod:
Watch it right away, for about a minute:
The container is restarted every 15 seconds and never gets to finish its start. After the third restart, the status
may also show CrashLoopBackOff: Kubernetes waits longer before each further restart, as in 2.3. Stop the watch with
Ctrl+C.
Add a startupProbe to the container in slow.yaml, right above the livenessProbe:
The probes of an existing Pod can’t be changed. Delete the Pod:
Create it again:
Watch it right away:
After about 25 seconds, the Pod is 1/1 without a single restart. Stop the watch with Ctrl+C.
Explanation
The liveness probe started checking right away. After three failed checks (15 seconds), the kubelet restarted the
container, long before it had finished starting. READY showed 1/1 the whole time: without a readiness probe,
Kubernetes considers a running container ready.
The startupProbe protects the start: as long as it hasn’t succeeded, the other probes wait and the Pod isn’t ready.
It allows up to 50 seconds (10 checks every 5 seconds), enough for a start of 20 seconds. After that, the liveness
probe takes over.
exec runs a command in the container; exit code 0 means success. sleep 20 & wait stands in for the slow start.
Together with trap 'exit 0' TERM, it lets the Pod stop at once when you delete it.
Step 4: Reserve and Limit Resources
Add resources to the container in web.yaml, right below the livenessProbe:
Apply the manifest:
Wait about 30 seconds and show how much CPU and memory the new Pods use:
If the new web Pods are missing or kubectl top reports that metrics aren’t available yet, wait a moment and
repeat the command.
Show the QoS class that Kubernetes assigned to the new Pods:
Explanation
m stands for millicores: 50m is 5 % of a CPU core. Mi stands for mebibytes.
requests are reserved for the container: the scheduler only places a Pod on a node with enough free room.
limits are the upper bound. Above its CPU limit, the container is slowed down, and above its memory limit, it’s
killed.
The QoS (quality of service) class follows from requests and limits: your Pods are Burstable, because their
requests are lower than their limits. When a node runs out of memory, Kubernetes evicts BestEffort Pods (without
any requests or limits) first and Guaranteed Pods (with requests equal to their limits) last.
kubectl top shows the current usage, collected by the metrics-server of your cluster. At rest, whoami needs far
less than it requested.
Step 5: Scale Automatically under Load
Create the file web-hpa.yaml with a HorizontalPodAutoscaler for the Deployment:
Create the autoscaler:
Wait about 20 seconds and list the autoscaler. By then, it shows the CPU usage of the Pods:
If TARGETS shows <unknown>, wait a moment and repeat the command.
Create the file load.yaml with a Pod that sends requests to the Service in an endless loop:
Start the load:
Watch the autoscaler right away:
After about a minute, the autoscaler has scaled the Deployment to 5 replicas. Stop the watch with Ctrl+C and stop the
load, because it keeps a CPU core of your machine busy:
Explanation
The HorizontalPodAutoscaler compares the CPU usage of the Pods with their CPU request: 50% means
25m per Pod on average. Under load, the Pods used far more, so it added Pods until it reached maxReplicas.
Without load, the autoscaler scales back to minReplicas, in your cluster about five minutes after the load
stopped. It waits that long on purpose, so that a short break in the load doesn’t remove Pods that are needed again a
moment later.
Optional Tasks
Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need. Open the solution if you get stuck.
Task 1: Over Budget
Give a new namespace budget a ResourceQuota that allows at most 200m of CPU requests (the
field is spec.hard.requests.cpu). Create the Deployment from web.yaml in it and scale it to 5
replicas. How many Pods start?
| Command | Effect |
|---|---|
kubectl scale deployment <name> --replicas=<number> | Changes the number of replicas |
kubectl events -n <namespace> | Lists the events of a namespace in time order |
Task 2: Wrong Number
Create a Pod with a readiness probe that checks the wrong port, 8080 instead of 80. What does Kubernetes report?
You know all the commands you need.
Clean Up
Delete all resources of this lab, since the following labs don’t need them:
--ignore-not-found skips the resources you already deleted, e.g. the load Pod. If you did Task 2, the Pod
wrong-port is deleted as well.