2.9. Health and Resources

In this lab, you tell Kubernetes when your application is ready and alive, how much CPU and memory it needs, and let it scale the application automatically under load.

Step 1: Check Readiness with a Probe

Create a directory for the files of this lab and change into it:

mkdir -p ~/labs/2.9
cd ~/labs/2.9

Create the file web.yaml in ~/labs/2.9. The whoami application from 2.6 answers on /health and lets you switch that answer to an error, so you can make a Pod unhealthy on purpose:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: whoami
          image: traefik/whoami:v1.12.0
          readinessProbe:
            httpGet:
              path: /health
              port: 80
            periodSeconds: 5

Create the file web-service.yaml with a Service for the Pods:

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

Create the Deployment:

kubectl apply -f web.yaml
deployment.apps/web created

Create the Service:

kubectl apply -f web-service.yaml
service/web-service created

List the Pods:

kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-64884bdbfb-7m4cs   1/1     Running   0          2s
web-64884bdbfb-brbfn   1/1     Running   0          2s

If the Pods aren’t 1/1 yet, wait a moment and repeat the command.

Forward a port to the first Pod (use a name from your output). The command keeps running:

kubectl port-forward pod/web-64884bdbfb-7m4cs 8080:80

In a second terminal, switch the health check of this Pod to the error code 500:

curl -X POST -d 500 http://localhost:8080/health

curl prints nothing. The first terminal shows Handling connection for 8080. Wait about 15 seconds and list the Pods again:

kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-64884bdbfb-7m4cs   0/1     Running   0          17s
web-64884bdbfb-brbfn   1/1     Running   0          17s

If the Pod is still 1/1, wait a few seconds and repeat the command.

Check which Pods the Service sends requests to:

kubectl describe service web-service
...
Endpoints:                10.42.1.72:80
...

Switch the health check back to 200:

curl -X POST -d 200 http://localhost:8080/health

Wait about 10 seconds and list the Pods again:

kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-64884bdbfb-7m4cs   1/1     Running   0          35s
web-64884bdbfb-brbfn   1/1     Running   0          35s

If the Pod is still 0/1, wait a moment and repeat the command.

Stop the port forwarding with Ctrl+C.

Explanation

The kubelet calls the readinessProbe every 5 seconds (periodSeconds). After three failed calls in a row (the default failureThreshold), it marks the Pod as not ready: READY 0/1. The Service removes a Pod that isn’t ready from its endpoints, so only the healthy Pod gets requests now. The unhealthy Pod keeps running and gets requests again as soon as the probe succeeds.

kubectl port-forward pod/... connects to the Pod directly, not through the Service. That’s why your curl still reached it.


Step 2: Restart a Broken Container

Add a livenessProbe to the container in web.yaml, right below the readinessProbe:

          livenessProbe:
            httpGet:
              path: /health
              port: 80
            periodSeconds: 5

In the first terminal, apply the manifest:

kubectl apply -f web.yaml
deployment.apps/web configured

List the new Pods:

kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-684bd5cf86-jg4qv   1/1     Running   0          4s
web-684bd5cf86-zjxw2   1/1     Running   0          3s

If the old Pods are still listed, wait a moment and repeat the command. Forward a port to one of the new Pods:

kubectl port-forward pod/web-684bd5cf86-jg4qv 8080:80

In the second terminal, switch its health check to 500 again. The command prints nothing:

curl -X POST -d 500 http://localhost:8080/health

Watch the Pods right away:

kubectl get pods --watch
NAME                   READY   STATUS    RESTARTS   AGE
web-684bd5cf86-jg4qv   1/1     Running   0          7s
web-684bd5cf86-zjxw2   1/1     Running   0          6s
web-684bd5cf86-jg4qv   0/1     Running   0          16s
web-684bd5cf86-jg4qv   0/1     Running   1 (1s ago)   21s
web-684bd5cf86-jg4qv   1/1     Running   1 (1s ago)   21s

After about 15 seconds, the liveness probe restarts the container (RESTARTS 1). The readiness probe calls the same endpoint and fails at about the same time, so the line with 0/1 and RESTARTS 0 may be missing in your output. Stop the watch and the port forwarding with Ctrl+C.

Explanation

The livenessProbe calls the same endpoint. After three failed calls, the kubelet restarts the container (RESTARTS 1). The new process of whoami starts healthy, so both probes succeed again.

ProbeQuestionWhen it fails
readinessProbeCan the Pod handle requests?The Service stops sending requests
livenessProbeIs the container still alive?The kubelet restarts the container

Step 3: Give a Slow Application Time to Start

Create the file slow.yaml with a Pod that needs 20 seconds to start. When its start is complete, it creates the file /tmp/started, and its liveness probe checks for that file with a command:

apiVersion: v1
kind: Pod
metadata:
  name: slow
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh", "-c", "trap 'exit 0' TERM; sleep 20 & wait; touch /tmp/started; while true; do sleep 1; done"]
      livenessProbe:
        exec:
          command: ["cat", "/tmp/started"]
        periodSeconds: 5

In the first terminal, create the Pod:

kubectl apply -f slow.yaml
pod/slow created

Watch it right away, for about a minute:

kubectl get pod slow --watch
NAME   READY   STATUS              RESTARTS     AGE
slow   0/1     ContainerCreating   0            0s
slow   1/1     Running             0            1s
slow   1/1     Running             1 (1s ago)   16s
slow   1/1     Running             2 (0s ago)   30s
slow   1/1     Running             3 (0s ago)   45s

The container is restarted every 15 seconds and never gets to finish its start. After the third restart, the status may also show CrashLoopBackOff: Kubernetes waits longer before each further restart, as in 2.3. Stop the watch with Ctrl+C.

Add a startupProbe to the container in slow.yaml, right above the livenessProbe:

      startupProbe:
        exec:
          command: ["cat", "/tmp/started"]
        periodSeconds: 5
        failureThreshold: 10

The probes of an existing Pod can’t be changed. Delete the Pod:

kubectl delete -f slow.yaml
pod "slow" deleted from default namespace

Create it again:

kubectl apply -f slow.yaml
pod/slow created

Watch it right away:

kubectl get pod slow --watch
NAME   READY   STATUS              RESTARTS   AGE
slow   0/1     ContainerCreating   0          0s
slow   0/1     Running             0          1s
slow   0/1     Running             0          25s
slow   1/1     Running             0          25s

After about 25 seconds, the Pod is 1/1 without a single restart. Stop the watch with Ctrl+C.

Explanation

The liveness probe started checking right away. After three failed checks (15 seconds), the kubelet restarted the container, long before it had finished starting. READY showed 1/1 the whole time: without a readiness probe, Kubernetes considers a running container ready.

The startupProbe protects the start: as long as it hasn’t succeeded, the other probes wait and the Pod isn’t ready. It allows up to 50 seconds (10 checks every 5 seconds), enough for a start of 20 seconds. After that, the liveness probe takes over.

exec runs a command in the container; exit code 0 means success. sleep 20 & wait stands in for the slow start. Together with trap 'exit 0' TERM, it lets the Pod stop at once when you delete it.


Step 4: Reserve and Limit Resources

Add resources to the container in web.yaml, right below the livenessProbe:

          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              cpu: 200m
              memory: 64Mi

Apply the manifest:

kubectl apply -f web.yaml

Wait about 30 seconds and show how much CPU and memory the new Pods use:

kubectl top pods
NAME                   CPU(cores)   MEMORY(bytes)
slow                   1m           0Mi
web-7f6894d8cf-n7k9d   1m           1Mi
web-7f6894d8cf-r4r27   1m           2Mi

If the new web Pods are missing or kubectl top reports that metrics aren’t available yet, wait a moment and repeat the command.

Show the QoS class that Kubernetes assigned to the new Pods:

kubectl describe pods -l app=web | grep QoS
QoS Class:                   Burstable
QoS Class:                   Burstable

Explanation

m stands for millicores: 50m is 5 % of a CPU core. Mi stands for mebibytes.

requests are reserved for the container: the scheduler only places a Pod on a node with enough free room. limits are the upper bound. Above its CPU limit, the container is slowed down, and above its memory limit, it’s killed.

The QoS (quality of service) class follows from requests and limits: your Pods are Burstable, because their requests are lower than their limits. When a node runs out of memory, Kubernetes evicts BestEffort Pods (without any requests or limits) first and Guaranteed Pods (with requests equal to their limits) last.

kubectl top shows the current usage, collected by the metrics-server of your cluster. At rest, whoami needs far less than it requested.


Step 5: Scale Automatically under Load

Create the file web-hpa.yaml with a HorizontalPodAutoscaler for the Deployment:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

Create the autoscaler:

kubectl apply -f web-hpa.yaml
horizontalpodautoscaler.autoscaling/web created

Wait about 20 seconds and list the autoscaler. By then, it shows the CPU usage of the Pods:

kubectl get hpa
NAME   REFERENCE        TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: 0%/50%   2         5         2          20s

If TARGETS shows <unknown>, wait a moment and repeat the command.

Create the file load.yaml with a Pod that sends requests to the Service in an endless loop:

apiVersion: v1
kind: Pod
metadata:
  name: load
spec:
  containers:
    - name: load
      image: busybox:1.37
      command: ["sh", "-c", "trap 'exit 0' TERM; while true; do wget -q -O /dev/null http://web-service; done"]

Start the load:

kubectl apply -f load.yaml
pod/load created

Watch the autoscaler right away:

kubectl get hpa web --watch
NAME   REFERENCE        TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   cpu: 0%/50%     2         5         2          21s
web    Deployment/web   cpu: 2%/50%     2         5         2          30s
web    Deployment/web   cpu: 156%/50%   2         5         2          45s
web    Deployment/web   cpu: 193%/50%   2         5         4          60s
web    Deployment/web   cpu: 70%/50%    2         5         5          75s
web    Deployment/web   cpu: 57%/50%    2         5         5          90s

After about a minute, the autoscaler has scaled the Deployment to 5 replicas. Stop the watch with Ctrl+C and stop the load, because it keeps a CPU core of your machine busy:

kubectl delete -f load.yaml

Explanation

The HorizontalPodAutoscaler compares the CPU usage of the Pods with their CPU request: 50% means 25m per Pod on average. Under load, the Pods used far more, so it added Pods until it reached maxReplicas.

Without load, the autoscaler scales back to minReplicas, in your cluster about five minutes after the load stopped. It waits that long on purpose, so that a short break in the load doesn’t remove Pods that are needed again a moment later.


Optional Tasks

Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need. Open the solution if you get stuck.

Task 1: Over Budget

Give a new namespace budget a ResourceQuota that allows at most 200m of CPU requests (the field is spec.hard.requests.cpu). Create the Deployment from web.yaml in it and scale it to 5 replicas. How many Pods start?

CommandEffect
kubectl scale deployment <name> --replicas=<number>Changes the number of replicas
kubectl events -n <namespace>Lists the events of a namespace in time order
Solution

Create the file quota.yaml in ~/labs/2.9:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: budget
spec:
  hard:
    requests.cpu: 200m

Create the namespace, the quota and the Deployment, and scale the Deployment:

kubectl create namespace budget
kubectl apply -f ~/labs/2.9/quota.yaml -n budget
kubectl apply -f ~/labs/2.9/web.yaml -n budget
kubectl scale deployment web -n budget --replicas=5

Wait a few seconds and list the Deployment:

kubectl get deployment web -n budget
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    4/5     4            4           30s

Only four Pods start: each requests 50m, and four of them use up the 200m of the quota. The events show why the fifth one is missing:

kubectl events -n budget
...
16s                 Warning   FailedCreate        ReplicaSet/web-7f6894d8cf   (combined from similar events): Error creating: pods "web-7f6894d8cf-f2j42" is forbidden: exceeded quota: budget, requested: requests.cpu=50m, used: requests.cpu=200m, limited: requests.cpu=200m

A ResourceQuota gives a namespace a budget, e.g. per team. Once it limits requests.cpu, every Pod in the namespace must request CPU: kubectl run without a request fails with must specify requests.cpu. A LimitRange gives such Pods a default request. Delete the namespace again. This takes about ten seconds:

kubectl delete namespace budget

Task 2: Wrong Number

Create a Pod with a readiness probe that checks the wrong port, 8080 instead of 80. What does Kubernetes report?

You know all the commands you need.

Solution

Create the file wrong-port.yaml in ~/labs/2.9:

apiVersion: v1
kind: Pod
metadata:
  name: wrong-port
spec:
  containers:
    - name: whoami
      image: traefik/whoami:v1.12.0
      readinessProbe:
        httpGet:
          path: /health
          port: 8080
        periodSeconds: 5

Create the Pod:

kubectl apply -f ~/labs/2.9/wrong-port.yaml

Wait about 20 seconds and list it:

kubectl get pod wrong-port
NAME         READY   STATUS    RESTARTS   AGE
wrong-port   0/1     Running   0          25s

The Pod runs but never becomes ready. Its events show the cause:

kubectl describe pod wrong-port
...
  Warning  Unhealthy  3s (x6 over 24s)  kubelet  spec.containers{whoami}: Readiness probe failed: Get "http://10.42.1.79:8080/health": dial tcp 10.42.1.79:8080: connect: connection refused

A probe on the wrong port or path is a common mistake: the application works, but the Service never sends it a request.


Clean Up

Delete all resources of this lab, since the following labs don’t need them:

kubectl delete -f ~/labs/2.9/ --ignore-not-found
pod "slow" deleted from default namespace
horizontalpodautoscaler.autoscaling "web" deleted from default namespace
service "web-service" deleted from default namespace
deployment.apps "web" deleted from default namespace

--ignore-not-found skips the resources you already deleted, e.g. the load Pod. If you did Task 2, the Pod wrong-port is deleted as well.


Complete files

~/labs/2.9/load.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: load
spec:
  containers:
    - name: load
      image: busybox:1.37
      command: ["sh", "-c", "trap 'exit 0' TERM; while true; do wget -q -O /dev/null http://web-service; done"]

~/labs/2.9/slow.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: slow
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh", "-c", "trap 'exit 0' TERM; sleep 20 & wait; touch /tmp/started; while true; do sleep 1; done"]
      startupProbe:
        exec:
          command: ["cat", "/tmp/started"]
        periodSeconds: 5
        failureThreshold: 10
      livenessProbe:
        exec:
          command: ["cat", "/tmp/started"]
        periodSeconds: 5

~/labs/2.9/web-hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

~/labs/2.9/web-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

~/labs/2.9/web.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: whoami
          image: traefik/whoami:v1.12.0
          readinessProbe:
            httpGet:
              path: /health
              port: 80
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 80
            periodSeconds: 5
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              cpu: 200m
              memory: 64Mi