2.10. Troubleshooting

In this lab, you find and fix four broken workloads with the troubleshooting tools of Kubernetes.

Step 1: Find the Broken Pods

A colleague deployed a small shop application, but nothing works. Create a directory for the files of this lab and change into it:

mkdir -p ~/labs/2.10
cd ~/labs/2.10

Create the file shop.yaml in ~/labs/2.10 with the colleague’s four Deployments. --- separates several manifests in one file. The containers simulate the applications with small shell scripts. You don’t need to understand them to find the errors:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
        - name: nginx
          image: ngnix:1.29
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: busybox:1.37
          command:
            - sh
            - -c
            - |
              trap 'exit 0' TERM
              if [ -z "$DATABASE_URL" ]; then
                echo "ERROR: DATABASE_URL is not set"
                exit 1
              fi
              echo "Connected to $DATABASE_URL"
              while true; do sleep 1; done
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: worker
  template:
    metadata:
      labels:
        app: worker
    spec:
      containers:
        - name: worker
          image: busybox:1.37
          command: ["sh", "-c", "trap 'exit 0' TERM; while true; do sleep 1; done"]
          resources:
            requests:
              memory: 64Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: report
spec:
  replicas: 1
  selector:
    matchLabels:
      app: report
  template:
    metadata:
      labels:
        app: report
    spec:
      containers:
        - name: report
          image: busybox:1.37
          command:
            - sh
            - -c
            - |
              trap 'exit 0' TERM
              data=$(head -c 100000000 /dev/zero | tr '\0' x)
              echo "Report ready" > /tmp/report.txt
              while true; do sleep 1; done
          resources:
            limits:
              memory: 64Mi

Create the Deployments:

kubectl apply -f shop.yaml

Wait about 30 seconds and list the Pods:

kubectl get pods
NAME                        READY   STATUS             RESTARTS      AGE
backend-6bf6455d89-kxkhp    0/1     Error              2 (29s ago)   30s
frontend-5ff6cd7dcf-9dr65   0/1     ImagePullBackOff   0             30s
report-85db4df785-m22rv     0/1     OOMKilled          2 (28s ago)   30s
worker-6cfd4568b7-29gnl     0/1     Pending            0             30s

Explanation

None of the Pods is ready. STATUS gives you the first clue for each of them:

StatusMeaning
ImagePullBackOff, ErrImagePullThe node can’t download the image
Error, CrashLoopBackOffThe container starts, ends with an error and is started again
PendingThe Pod has no node yet
OOMKilledThe container used more memory than its limit and was killed

The statuses change while Kubernetes retries, e.g. between ErrImagePull and ImagePullBackOff or between Error and CrashLoopBackOff, so yours may differ from the output above.


Step 2: Read the Events

Show the details of the Pod of the Deployment frontend. -l selects it by its label, so you don’t need its name:

kubectl describe pod -l app=frontend
...
Events:
  ...
  Warning  Failed     19s (x3 over 59s)  kubelet            spec.containers{nginx}: Failed to pull image "ngnix:1.29": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/ngnix:1.29": failed to resolve reference "docker.io/library/ngnix:1.29": docker.io/library/ngnix:1.29: not found
  ...

Show the details of the Pod of the Deployment worker:

kubectl describe pod -l app=worker
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  60s   default-scheduler  0/3 nodes are available: 3 Insufficient memory. no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Explanation

The events at the end of kubectl describe record what Kubernetes tried and why it failed:

  • frontend: there is no image ngnix (a typo of nginx), so the registry answers not found.
  • worker: none of the three nodes has 64 GiB of memory free for the request, so the scheduler can’t place the Pod. The rest of the message says that evicting other Pods wouldn’t help either.

Step 3: Read the Logs and the Last State

Show the output of the Deployment backend:

kubectl logs deployment/backend
ERROR: DATABASE_URL is not set

Show the details of the Pod of the Deployment report:

kubectl describe pod -l app=report
...
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
...

Explanation

kubectl describe would only tell you that the container of backend keeps ending. The cause is in its own output: the application expects the environment variable DATABASE_URL. deployment/backend lets kubectl logs pick a Pod of the Deployment for you, like job/backup in 2.5. kubectl logs shows the output of the last run, even if the container has already ended. Once the container has started again, kubectl logs --previous shows the output of the run before.

Last State shows why the previous run of report ended: it used more than its memory limit of 64 MiB, so the kernel killed it. Exit code 137 means that the process was killed; Reason: OOMKilled tells you it was because of the memory limit.


Step 4: Fix the Workloads

Fix the four errors in shop.yaml. Correct the image of frontend:

          image: nginx:1.29

Add the environment variable to the container of backend, right below its image:

          env:
            - name: DATABASE_URL
              value: postgres://db:5432/shop

Lower the memory request of worker:

            requests:
              memory: 64Mi

Raise the memory limit of report:

            limits:
              memory: 256Mi

Apply the manifest:

kubectl apply -f shop.yaml
deployment.apps/frontend configured
deployment.apps/backend configured
deployment.apps/worker configured
deployment.apps/report configured

List the Pods:

kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
backend-7c866b666b-w8ft7    1/1     Running   0          3s
frontend-7b67bcfc69-lx6bm   1/1     Running   0          3s
report-d758d689d-tvmvm      1/1     Running   0          3s
worker-885cfc9f5-64n9m      1/1     Running   0          3s

If not all Pods are 1/1 yet, or the old Pods are still listed, wait a moment and repeat the command. Wait about 20 seconds more and show how much memory the Pods use:

kubectl top pods
NAME                        CPU(cores)   MEMORY(bytes)
backend-7c866b666b-w8ft7    2m           0Mi
frontend-7b67bcfc69-lx6bm   0m           3Mi
report-d758d689d-tvmvm      3m           96Mi
worker-885cfc9f5-64n9m      1m           0Mi

If kubectl top reports that metrics aren’t available yet, wait a moment and repeat the command.

Explanation

env sets a single environment variable directly in the manifest. For a real password, you’d take it from a Secret as in 2.7. kubectl top confirms the last fix: report needs about 96 MiB, more than its old limit of 64 MiB.


Step 5: Look Inside a Running Container

Run a command inside the container of backend:

kubectl exec deployment/backend -- printenv DATABASE_URL
postgres://db:5432/shop

Open a shell in the container:

kubectl exec -it deployment/backend -- sh
/ #

In the shell, list the processes of the container:

ps
PID   USER     TIME  COMMAND
    1 root      0:00 sh -c trap 'exit 0' TERM if [ -z "$DATABASE_URL" ]; then
  120 root      0:00 sh
  129 root      0:00 sleep 1
  130 root      0:00 ps

Leave the shell with exit.

Explanation

kubectl exec runs a command in a running container, as if you were logged in to it. Everything after -- is the command. With -it and a shell, you work in the container interactively, for example to check files, environment variables or the network from the application’s point of view.

ps shows the sh -c script from shop.yaml as PID 1, next to the shell you opened: the container runs only its application, no other services.


Optional Tasks

Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need. Open the solution if you get stuck.

Task 1: No Shell? No Problem

Many images contain only the application and no shell. Create the file api.yaml in ~/labs/2.10 with a Pod api that runs the whoami application from 2.6, and try to open a shell in it. Then look at its processes with a debug container.

CommandEffect
kubectl debug -it <pod> --image=<image> --target=<container>Adds a temporary container to the Pod that sees the processes of the given container
Solution

Create the file api.yaml in ~/labs/2.10:

apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  containers:
    - name: api
      image: traefik/whoami:v1.12.0

Create the Pod:

kubectl apply -f ~/labs/2.10/api.yaml

Wait a few seconds and try to open a shell:

kubectl exec -it api -- sh
Error: Internal error occurred: ... exec: "sh": executable file not found in $PATH

The image has no shell. Add a debug container with busybox instead:

kubectl debug -it api --image=busybox:1.37 --target=api
Targeting container "api". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-mbnxl.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
/ #

In the shell of the debug container, list the processes:

ps
PID   USER     TIME  COMMAND
    1 root      0:00 /whoami
...

Call the application:

wget -q -O - http://localhost
Hostname: api
...

The debug container brings its own tools and sees the processes and the network of the container api. Leave it with exit. kubectl then prints Session ended, …, which you can ignore.

Task 2: Take It Home

The Deployment report writes the file /tmp/report.txt. Copy it from the container to your machine.

CommandEffect
kubectl cp <pod>:<path in container> <local path>Copies a file from a container to your machine
Solution

Find the name of the Pod:

kubectl get pods -l app=report
NAME                     READY   STATUS    RESTARTS   AGE
report-d758d689d-tvmvm   1/1     Running   0          3m

Copy the file with the Pod name from your output:

kubectl cp report-d758d689d-tvmvm:/tmp/report.txt report.txt
tar: removing leading '/' from member names

Show the copied file:

cat report.txt
Report ready

kubectl cp uses tar inside the container, so the image has to contain it. The message about the leading / is harmless. The other direction works as well: kubectl cp report.txt <pod>:/tmp/ copies a file into the container.


Clean Up

Delete all resources of this lab, since the following labs don’t need them:

kubectl delete -f ~/labs/2.10/
deployment.apps "frontend" deleted from default namespace
deployment.apps "backend" deleted from default namespace
deployment.apps "worker" deleted from default namespace
deployment.apps "report" deleted from default namespace

If you did Task 1, the Pod api is deleted as well.


Complete files

~/labs/2.10/shop.yaml with all four fixes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
        - name: nginx
          image: nginx:1.29
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: busybox:1.37
          env:
            - name: DATABASE_URL
              value: postgres://db:5432/shop
          command:
            - sh
            - -c
            - |
              trap 'exit 0' TERM
              if [ -z "$DATABASE_URL" ]; then
                echo "ERROR: DATABASE_URL is not set"
                exit 1
              fi
              echo "Connected to $DATABASE_URL"
              while true; do sleep 1; done
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: worker
  template:
    metadata:
      labels:
        app: worker
    spec:
      containers:
        - name: worker
          image: busybox:1.37
          command: ["sh", "-c", "trap 'exit 0' TERM; while true; do sleep 1; done"]
          resources:
            requests:
              memory: 64Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: report
spec:
  replicas: 1
  selector:
    matchLabels:
      app: report
  template:
    metadata:
      labels:
        app: report
    spec:
      containers:
        - name: report
          image: busybox:1.37
          command:
            - sh
            - -c
            - |
              trap 'exit 0' TERM
              data=$(head -c 100000000 /dev/zero | tr '\0' x)
              echo "Report ready" > /tmp/report.txt
              while true; do sleep 1; done
          resources:
            limits:
              memory: 256Mi