2.5. Beyond Deployments

In this lab, you run a Pod on every node with a DaemonSet, give Pods a stable identity with a StatefulSet and run tasks once with a Job or on a schedule with a CronJob, which you then pause.

Step 1: Run a Pod on Every Node

Create a directory for the files of this lab and change into it:

mkdir -p ~/labs/2.5
cd ~/labs/2.5

Create the file node-exporter.yaml in ~/labs/2.5. node-exporter is a monitoring agent that collects the CPU, memory and disk metrics of its node:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      containers:
        - name: node-exporter
          image: prom/node-exporter:v1.12.1

Create the DaemonSet:

kubectl apply -f node-exporter.yaml
daemonset.apps/node-exporter created

List the DaemonSets:

kubectl get daemonsets
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-exporter   3         3         3       3            3           <none>          3s

If READY isn’t 3 yet, the Pods are still starting. Wait a moment and repeat the command.

Show on which nodes its Pods run. -o wide adds the column NODE:

kubectl get pods -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
node-exporter-7gbht   1/1     Running   0          3s    10.42.1.16   k3d-labs-agent-1    <none>           <none>
node-exporter-sspxt   1/1     Running   0          3s    10.42.2.15   k3d-labs-server-0   <none>           <none>
node-exporter-vtcnb   1/1     Running   0          3s    10.42.0.36   k3d-labs-agent-0    <none>           <none>

Explanation

A DaemonSet runs exactly one Pod on every node, so its manifest has no replicas. DESIRED 3 is the number of nodes in your cluster. Agents that every node needs (for monitoring, logging or networking) run as DaemonSets.


Step 2: Give Pods a Stable Identity

Create the file redis.yaml with a StatefulSet that runs three instances of the database Redis:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:8.2

Create the StatefulSet:

kubectl apply -f redis.yaml
statefulset.apps/redis created

Watch its Pods right away:

kubectl get pods -l app=redis --watch
NAME      READY   STATUS              RESTARTS   AGE
redis-0   0/1     ContainerCreating   0          0s
redis-0   1/1     Running             0          4s
redis-1   0/1     Pending             0          0s
redis-1   0/1     ContainerCreating   0          0s
redis-1   1/1     Running             0          4s
redis-2   0/1     Pending             0          0s
redis-2   0/1     ContainerCreating   0          0s
redis-2   1/1     Running             0          5s

Stop the watch with Ctrl+C and delete the Pod redis-1:

kubectl delete pod redis-1
pod "redis-1" deleted from default namespace

List the Pods again:

kubectl get pods -l app=redis
NAME      READY   STATUS              RESTARTS   AGE
redis-0   1/1     Running             0          46s
redis-1   0/1     ContainerCreating   0          1s
redis-2   1/1     Running             0          38s

Explanation

The manifest looks like a Deployment: only the kind differs. But a StatefulSet numbers its Pods instead of giving them random names. It starts them one after another: redis-1 only starts once redis-0 runs. A replaced Pod gets the same name back, here a new redis-1.

Databases and other clustered applications rely on this stable identity, e.g. a replicated database can always make redis-0 its primary. The StatefulSet only provides the identity: the three Redis instances don’t replicate any data between each other by themselves.


Step 3: Run a Task Once

Create the file backup.yaml with a Job that simulates a backup:

apiVersion: batch/v1
kind: Job
metadata:
  name: backup
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: backup
          image: busybox:1.37
          command: ["sh", "-c", "echo Creating backup...; sleep 5; echo Backup done"]

Create the Job:

kubectl apply -f backup.yaml

Wait about ten seconds and list the Jobs:

kubectl get jobs
NAME     STATUS     COMPLETIONS   DURATION   AGE
backup   Complete   1/1           7s         7s

If STATUS isn’t Complete yet, wait a moment and repeat the command.

Show the output of the Job:

kubectl logs job/backup
Creating backup...
Backup done

Explanation

A Job runs a Pod until its task has ended successfully. COMPLETIONS 1/1 means that the task ran once and succeeded. A Job only accepts restartPolicy: Never or OnFailure. With Always, the task would never end. Unlike a Deployment, a Job needs no selector: it labels its Pods itself, with job-name.

The finished Pod stays with the status Completed, so you can still read its output. job/backup tells kubectl logs to find the Pod of the Job for you.


Step 4: Run a Task on a Schedule

Create the file backup-cron.yaml with a CronJob that runs the backup every minute:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-cron
spec:
  schedule: "* * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: busybox:1.37
              command: ["sh", "-c", "date; echo Backup done"]

Create the CronJob:

kubectl apply -f backup-cron.yaml
cronjob.batch/backup-cron created

List the CronJobs:

kubectl get cronjobs
NAME          SCHEDULE    TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
backup-cron   * * * * *   <none>     False     0        <none>          0s

Wait two minutes and list the Jobs:

kubectl get jobs
NAME                   STATUS     COMPLETIONS   DURATION   AGE
backup                 Complete   1/1           7s         2m28s
backup-cron-29864998   Complete   1/1           3s         101s
backup-cron-29864999   Complete   1/1           3s         41s

If the newest Job isn’t Complete yet, wait a moment and repeat the command.

Show the output of the newest Job. Use a name from your output:

kubectl logs job/backup-cron-29864999
Tue Oct 13 13:59:00 UTC 2026
Backup done

Explanation

A CronJob creates a new Job on every scheduled time. jobTemplate describes that Job in the same way as the Job in Step 3. The number in the name of each Job is its scheduled time, in minutes since 1970.

The schedule uses the cron format with five fields: minute, hour, day of month, month and day of week. * * * * * means every minute, 0 2 * * * would mean every night at 2:00. Without the field timeZone, the times are in the time zone of the control plane (UTC in your cluster, like the time in the output of the Job).

The CronJob keeps the last three successful Jobs and deletes older ones.


Step 5: Pause the Schedule

The CronJob creates a new Job every minute. Pause it with kubectl patch, which changes single fields of a resource in the cluster:

kubectl patch cronjob backup-cron -p '{"spec":{"suspend":true}}'
cronjob.batch/backup-cron patched

List the CronJobs:

kubectl get cronjobs
NAME          SCHEDULE    TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
backup-cron   * * * * *   <none>     True      0        19s             2m41s

Apply the manifest again:

kubectl apply -f backup-cron.yaml
cronjob.batch/backup-cron unchanged

Explanation

-p passes the fields to change as JSON, here spec.suspend. kubectl patch merges them into the CronJob and leaves all other fields as they are. A suspended CronJob creates no new Jobs.

Like kubectl edit in 2.4, kubectl patch is imperative: it changes the cluster, but not your manifest. Your manifest has no field suspend, and kubectl apply only sets the fields that the manifest contains. That’s why it reported unchanged, and the CronJob stays paused, although nobody can tell from the manifest. Use kubectl patch only for a quick fix or a test, e.g. to pause a backup during maintenance. If the change should stay, write suspend: true into the manifest.


Optional Tasks

Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need. Open the solution if you get stuck.

Task 1: New Node, New Pod

Add a fourth node to the cluster. What does the DaemonSet do? Remove the node again afterwards.

CommandEffect
k3d node create <name> --cluster labs --role agentAdds an agent node. Its container is named k3d-<name>-0
k3d node delete <container>Removes the container of a node
kubectl delete node <name>Removes the Node from Kubernetes
Solution

Add the node extra:

k3d node create extra --cluster labs --role agent

Wait about 20 seconds, until the network of the new node is ready, and list the Pods of the DaemonSet:

kubectl get pods -l app=node-exporter -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
node-exporter-7gbht   1/1     Running   0          10m   10.42.1.16   k3d-labs-agent-1    <none>           <none>
node-exporter-jgr9j   1/1     Running   0          5s    10.42.3.2    k3d-extra-0         <none>           <none>
node-exporter-sspxt   1/1     Running   0          10m   10.42.2.15   k3d-labs-server-0   <none>           <none>
node-exporter-vtcnb   1/1     Running   0          10m   10.42.0.36   k3d-labs-agent-0    <none>           <none>

If the Pod on k3d-extra-0 isn’t Running yet, wait a moment and repeat the command.

The DaemonSet started a Pod on the new node by itself. Remove the node. k3d node delete only removes the container, so delete the Node as well:

k3d node delete k3d-extra-0
kubectl delete node k3d-extra-0

Within about a minute, kubectl get pods -l app=node-exporter -o wide shows three Pods again.

Task 2: Last In, First Out

Scale the StatefulSet down to one replica and back up to three. In which order do the Pods stop and start?

CommandEffect
kubectl scale statefulset <name> --replicas=<number>Changes the number of replicas
Solution

Watch the Pods in one terminal:

kubectl get pods -l app=redis --watch

Scale down in a second terminal:

kubectl scale statefulset redis --replicas=1

The watch in the first terminal shows:

redis-2   1/1     Terminating   0          3m6s
redis-2   0/1     Completed     0          3m7s
redis-1   1/1     Terminating   0          2m30s
redis-1   0/1     Completed     0          2m31s

Scale up again:

kubectl scale statefulset redis --replicas=3

The watch in the first terminal shows:

redis-1   0/1     ContainerCreating   0          0s
redis-1   1/1     Running             0          1s
redis-2   0/1     Pending             0          0s
redis-2   0/1     ContainerCreating   0          0s
redis-2   1/1     Running             0          1s

The StatefulSet stops its Pods in reverse order (the highest number first) and starts them in ascending order. redis-0 keeps running all the time.

redis.yaml still says replicas: 3, so the cluster matches the file again.

Task 3: Three Strikes

Create a Job whose task always fails, and limit the number of retries with backoffLimit: 2 in its spec. How often does Kubernetes try, and what is the result?

You know all the commands you need. A container fails when its command ends with an exit code other than 0, e.g. exit 1.

Solution

Create the file three-strikes.yaml in ~/labs/2.5:

apiVersion: batch/v1
kind: Job
metadata:
  name: three-strikes
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: backup
          image: busybox:1.37
          command: ["sh", "-c", "echo Backup failed; exit 1"]

Create the Job:

kubectl apply -f ~/labs/2.5/three-strikes.yaml

Wait about 40 seconds and list the Job and its Pods:

kubectl get job three-strikes
NAME            STATUS   COMPLETIONS   DURATION   AGE
three-strikes   Failed   0/1           40s        40s

List its Pods by the label job-name:

kubectl get pods -l job-name=three-strikes
NAME                  READY   STATUS   RESTARTS   AGE
three-strikes-5ftlx   0/1     Error    0          29s
three-strikes-5xl8q   0/1     Error    0          9s
three-strikes-q8btp   0/1     Error    0          40s

Kubernetes tried three times: the first run plus two retries. It waits a little longer before every retry, and after the last one, it marks the Job as Failed. Without backoffLimit, a Job retries up to six times.


Clean Up

Delete all resources of this lab, since the following labs don’t need them:

kubectl delete -f ~/labs/2.5/
cronjob.batch "backup-cron" deleted from default namespace
job.batch "backup" deleted from default namespace
daemonset.apps "node-exporter" deleted from default namespace
statefulset.apps "redis" deleted from default namespace

Kubernetes deletes their Pods and the Jobs of the CronJob along with them. If you did Task 3, the Job three-strikes is deleted as well.