In this lab, you find and fix four broken workloads with the troubleshooting tools of Kubernetes.
Step 1: Find the Broken Pods
A colleague deployed a small shop application, but nothing works. Create a directory for the files of this lab and
change into it:
mkdir -p ~/labs/2.10
cd ~/labs/2.10
Create the file shop.yaml in ~/labs/2.10 with the colleague’s four Deployments. --- separates
several manifests in one file. The containers simulate the applications with small shell scripts. You don’t need to
understand them to find the errors:
apiVersion:apps/v1kind:Deploymentmetadata:name:frontendspec:replicas:1selector:matchLabels:app:frontendtemplate:metadata:labels:app:frontendspec:containers:- name:nginximage:ngnix:1.29---apiVersion:apps/v1kind:Deploymentmetadata:name:backendspec:replicas:1selector:matchLabels:app:backendtemplate:metadata:labels:app:backendspec:containers:- name:backendimage:busybox:1.37command:- sh- -c- | trap 'exit 0' TERM
if [ -z "$DATABASE_URL" ]; then
echo "ERROR: DATABASE_URL is not set"
exit 1
fi
echo "Connected to $DATABASE_URL"
while true; do sleep 1; done---apiVersion:apps/v1kind:Deploymentmetadata:name:workerspec:replicas:1selector:matchLabels:app:workertemplate:metadata:labels:app:workerspec:containers:- name:workerimage:busybox:1.37command:["sh","-c","trap 'exit 0' TERM; while true; do sleep 1; done"]resources:requests:memory:64Gi---apiVersion:apps/v1kind:Deploymentmetadata:name:reportspec:replicas:1selector:matchLabels:app:reporttemplate:metadata:labels:app:reportspec:containers:- name:reportimage:busybox:1.37command:- sh- -c- | trap 'exit 0' TERM
data=$(head -c 100000000 /dev/zero | tr '\0' x)
echo "Report ready" > /tmp/report.txt
while true; do sleep 1; doneresources:limits:memory:64Mi
Create the Deployments:
kubectl apply -f shop.yaml
Wait about 30 seconds and list the Pods:
kubectl get pods
NAME READY STATUS RESTARTS AGE
backend-6bf6455d89-kxkhp 0/1 Error 2 (29s ago) 30s
frontend-5ff6cd7dcf-9dr65 0/1 ImagePullBackOff 0 30s
report-85db4df785-m22rv 0/1 OOMKilled 2 (28s ago) 30s
worker-6cfd4568b7-29gnl 0/1 Pending 0 30s
Explanation
None of the Pods is ready. STATUS gives you the first clue for each of them:
Status
Meaning
ImagePullBackOff, ErrImagePull
The node can’t download the image
Error, CrashLoopBackOff
The container starts, ends with an error and is started again
Pending
The Pod has no node yet
OOMKilled
The container used more memory than its limit and was killed
The statuses change while Kubernetes retries, e.g. between ErrImagePull and ImagePullBackOff or between Error
and CrashLoopBackOff, so yours may differ from the output above.
Step 2: Read the Events
Show the details of the Pod of the Deployment frontend. -l selects it by its label, so you don’t need its name:
kubectl describe pod -l app=frontend
...
Events:
...
Warning Failed 19s (x3 over 59s) kubelet spec.containers{nginx}: Failed to pull image "ngnix:1.29": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/ngnix:1.29": failed to resolve reference "docker.io/library/ngnix:1.29": docker.io/library/ngnix:1.29: not found
...
Show the details of the Pod of the Deployment worker:
kubectl describe pod -l app=worker
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 60s default-scheduler 0/3 nodes are available: 3 Insufficient memory. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Explanation
The events at the end of kubectl describe record what Kubernetes tried and why it failed:
frontend: there is no image ngnix (a typo of nginx), so the registry answers not found.
worker: none of the three nodes has 64 GiB of memory free for the request, so the scheduler can’t place the Pod.
The rest of the message says that evicting other Pods wouldn’t help either.
Step 3: Read the Logs and the Last State
Show the output of the Deployment backend:
kubectl logs deployment/backend
ERROR: DATABASE_URL is not set
Show the details of the Pod of the Deployment report:
kubectl describe pod -l app=report
...
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
...
Explanation
kubectl describe would only tell you that the container of backend keeps ending. The cause is in its own output:
the application expects the environment variable DATABASE_URL. deployment/backend lets kubectl logs pick a Pod
of the Deployment for you, like job/backup in 2.5. kubectl logs shows the output of the last run, even if the
container has already ended. Once the container has started again, kubectl logs --previous shows the output of the
run before.
Last State shows why the previous run of report ended: it used more than its memory limit of 64 MiB, so the
kernel killed it. Exit code 137 means that the process was killed; Reason: OOMKilled tells you it was because of
the memory limit.
Step 4: Fix the Workloads
Fix the four errors in shop.yaml. Correct the image of frontend:
image:nginx:1.29
Add the environment variable to the container of backend, right below its image:
NAME READY STATUS RESTARTS AGE
backend-7c866b666b-w8ft7 1/1 Running 0 3s
frontend-7b67bcfc69-lx6bm 1/1 Running 0 3s
report-d758d689d-tvmvm 1/1 Running 0 3s
worker-885cfc9f5-64n9m 1/1 Running 0 3s
If not all Pods are 1/1 yet, or the old Pods are still listed, wait a moment and repeat the command. Wait about 20
seconds more and show how much memory the Pods use:
If kubectl top reports that metrics aren’t available yet, wait a moment and repeat the command.
Explanation
env sets a single environment variable directly in the manifest. For a real password, you’d take it from a Secret
as in 2.7. kubectl top confirms the last fix: report needs about 96 MiB, more than its old limit of 64 MiB.
In the shell, list the processes of the container:
ps
PID USER TIME COMMAND
1 root 0:00 sh -c trap 'exit 0' TERM if [ -z "$DATABASE_URL" ]; then
120 root 0:00 sh
129 root 0:00 sleep 1
130 root 0:00 ps
Leave the shell with exit.
Explanation
kubectl exec runs a command in a running container, as if you were logged in to it. Everything after -- is the
command. With -it and a shell, you work in the container interactively, for example to check files, environment
variables or the network from the application’s point of view.
ps shows the sh -c script from shop.yaml as PID 1, next to the shell you opened: the container runs
only its application, no other services.
Optional Tasks
Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need.
Open the solution if you get stuck.
Task 1: No Shell? No Problem
Many images contain only the application and no shell. Create the file api.yaml in ~/labs/2.10 with
a Pod api that runs the whoami application from 2.6, and try to open a shell in it. Then look at its processes with
a debug container.
Error: Internal error occurred: ... exec: "sh": executable file not found in $PATH
The image has no shell. Add a debug container with busybox instead:
kubectl debug -it api --image=busybox:1.37 --target=api
Targeting container "api". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-mbnxl.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
/ #
In the shell of the debug container, list the processes:
ps
PID USER TIME COMMAND
1 root 0:00 /whoami
...
Call the application:
wget -q -O - http://localhost
Hostname: api
...
The debug container brings its own tools and sees the processes and the network of the container api. Leave it with
exit. kubectl then prints Session ended, …, which you can ignore.
Task 2: Take It Home
The Deployment report writes the file /tmp/report.txt. Copy it from the container to your machine.
Command
Effect
kubectl cp <pod>:<path in container> <local path>
Copies a file from a container to your machine
Solution
Find the name of the Pod:
kubectl get pods -l app=report
NAME READY STATUS RESTARTS AGE
report-d758d689d-tvmvm 1/1 Running 0 3m
kubectl cp uses tar inside the container, so the image has to contain it. The message about the leading / is
harmless. The other direction works as well: kubectl cp report.txt <pod>:/tmp/ copies a file into the container.
Clean Up
Delete all resources of this lab, since the following labs don’t need them:
kubectl delete -f ~/labs/2.10/
deployment.apps "frontend" deleted from default namespace
deployment.apps "backend" deleted from default namespace
deployment.apps "worker" deleted from default namespace
deployment.apps "report" deleted from default namespace
If you did Task 1, the Pod api is deleted as well.
Complete files
~/labs/2.10/shop.yaml with all four fixes:
apiVersion:apps/v1kind:Deploymentmetadata:name:frontendspec:replicas:1selector:matchLabels:app:frontendtemplate:metadata:labels:app:frontendspec:containers:- name:nginximage:nginx:1.29---apiVersion:apps/v1kind:Deploymentmetadata:name:backendspec:replicas:1selector:matchLabels:app:backendtemplate:metadata:labels:app:backendspec:containers:- name:backendimage:busybox:1.37env:- name:DATABASE_URLvalue:postgres://db:5432/shopcommand:- sh- -c- | trap 'exit 0' TERM
if [ -z "$DATABASE_URL" ]; then
echo "ERROR: DATABASE_URL is not set"
exit 1
fi
echo "Connected to $DATABASE_URL"
while true; do sleep 1; done---apiVersion:apps/v1kind:Deploymentmetadata:name:workerspec:replicas:1selector:matchLabels:app:workertemplate:metadata:labels:app:workerspec:containers:- name:workerimage:busybox:1.37command:["sh","-c","trap 'exit 0' TERM; while true; do sleep 1; done"]resources:requests:memory:64Mi---apiVersion:apps/v1kind:Deploymentmetadata:name:reportspec:replicas:1selector:matchLabels:app:reporttemplate:metadata:labels:app:reportspec:containers:- name:reportimage:busybox:1.37command:- sh- -c- | trap 'exit 0' TERM
data=$(head -c 100000000 /dev/zero | tr '\0' x)
echo "Report ready" > /tmp/report.txt
while true; do sleep 1; doneresources:limits:memory:256Mi