2.8. Storage

In this lab, you see how long data lives in a volume: as long as its Pod, or beyond it with a PersistentVolumeClaim.

Step 1: Lose Data with an emptyDir

Create a directory for the files of this lab and change into it:

mkdir -p ~/labs/2.8
cd ~/labs/2.8

Create the file scratch.yaml in ~/labs/2.8. Its init container adds a line to the file /data/log.txt, and its main container prints the file:

apiVersion: v1
kind: Pod
metadata:
  name: scratch
spec:
  restartPolicy: Never
  initContainers:
    - name: writer
      image: busybox:1.37
      command: ["sh", "-c", "echo Started at $(date) >> /data/log.txt"]
      volumeMounts:
        - name: data
          mountPath: /data
  containers:
    - name: reader
      image: busybox:1.37
      command: ["cat", "/data/log.txt"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      emptyDir: {}

Create the Pod:

kubectl apply -f scratch.yaml

Wait a few seconds and show the output of the container reader:

kubectl logs scratch -c reader
Started at Tue Oct 13 14:35:18 UTC 2026

If kubectl logs reports that the container reader is waiting to start, wait a moment and repeat the command.

Delete the Pod and create it again:

kubectl delete -f scratch.yaml
kubectl apply -f scratch.yaml

Wait a few seconds and show the output again:

kubectl logs scratch -c reader
Started at Tue Oct 13 14:35:22 UTC 2026

Explanation

Both containers mount the same volume at /data, so the container reader sees the file the init container wrote. An emptyDir volume is created empty together with its Pod and deleted together with it. That’s why the new Pod shows a single line again. The first line is gone.


Step 2: Request Storage

Create the file data-pvc.yaml with a PersistentVolumeClaim (PVC) for 1 GiB of storage:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Create the claim:

kubectl apply -f data-pvc.yaml
persistentvolumeclaim/data created

List the PVCs:

kubectl get pvc
NAME   STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data   Pending                                      local-path     <unset>                 0s

The status Pending is expected. The explanation below says why.

List the StorageClasses of your cluster:

kubectl get storageclass
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  4h25m

Explanation

A PersistentVolumeClaim (PVC) requests storage without saying where it comes from. That’s the job of the StorageClass: k3s ships local-path, which stores the data in a directory on a node. It’s the default, so the PVC uses it without asking for it.

WaitForFirstConsumer explains the status Pending: local-path only creates the storage once a Pod uses the PVC, on the node where that Pod runs. ReadWriteOnce means that only one node at a time can use the storage.


Step 3: Keep Data with a PersistentVolumeClaim

Create the file keeper.yaml: the Pod from Step 1, renamed to keeper, whose volume at the end now refers to the PVC instead of an emptyDir:

apiVersion: v1
kind: Pod
metadata:
  name: keeper
spec:
  restartPolicy: Never
  initContainers:
    - name: writer
      image: busybox:1.37
      command: ["sh", "-c", "echo Started at $(date) >> /data/log.txt"]
      volumeMounts:
        - name: data
          mountPath: /data
  containers:
    - name: reader
      image: busybox:1.37
      command: ["cat", "/data/log.txt"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data

Create the Pod:

kubectl apply -f keeper.yaml

Wait a few seconds and list the PVC and the PersistentVolumes:

kubectl get pvc
NAME   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data   Bound    pvc-f6a1add7-a5a1-4a04-a905-52dd2c299e45   1Gi        RWO            local-path     <unset>                 11s

If STATUS isn’t Bound yet, local-path is still creating the storage. Wait a moment and repeat the command.

kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM          STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-f6a1add7-a5a1-4a04-a905-52dd2c299e45   1Gi        RWO            Delete           Bound    default/data   local-path     <unset>                          6s

Delete the Pod and create it again:

kubectl delete -f keeper.yaml
kubectl apply -f keeper.yaml

Wait a few seconds and show the output of the container reader:

kubectl logs keeper -c reader
Started at Tue Oct 13 14:35:33 UTC 2026
Started at Tue Oct 13 14:35:37 UTC 2026

If kubectl logs reports that the container reader is waiting to start, wait a moment and repeat the command.

Explanation

As soon as the first Pod used the PVC, local-path created a PersistentVolume (PV) and bound the PVC to it. That’s dynamic provisioning. The PV lives independently of the Pods: the new Pod found the line of the first one and added its own.

The PV is a directory on one node, so every Pod that uses the PVC runs on that node. The RECLAIM POLICY Delete means: when you delete the PVC, Kubernetes deletes the PV and its data as well.


Optional Tasks

Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need. Open the solution if you get stuck.

Task 1: Where Is My Data?

Find the file log.txt of the PVC data on the node. local-path stores its volumes in /var/lib/rancher/k3s/storage.

CommandEffect
docker exec <container> <command>Runs a command inside a running container
Solution

Find the node of the Pod keeper:

kubectl get pod keeper -o wide
NAME     READY   STATUS      RESTARTS   AGE   IP           NODE               NOMINATED NODE   READINESS GATES
keeper   0/1     Completed   0          4s    10.42.1.52   k3d-labs-agent-1   <none>           <none>

The node is the Docker container of the same name. List the volumes on it:

docker exec k3d-labs-agent-1 ls /var/lib/rancher/k3s/storage
pvc-f6a1add7-a5a1-4a04-a905-52dd2c299e45_default_data

The directory is named after the PV, the namespace and the PVC. Show the file (use your directory name):

docker exec k3d-labs-agent-1 cat /var/lib/rancher/k3s/storage/pvc-f6a1add7-a5a1-4a04-a905-52dd2c299e45_default_data/log.txt
Started at Tue Oct 13 14:35:33 UTC 2026
Started at Tue Oct 13 14:35:37 UTC 2026

Task 2: Wrong Class

Create a PVC that asks for the StorageClass fast with the field storageClassName. What happens, and where do you find the cause?

You know all the commands you need.

Solution

Create the file wrong-class.yaml in ~/labs/2.8:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wrong-class
spec:
  storageClassName: fast
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Create the PVC:

kubectl apply -f ~/labs/2.8/wrong-class.yaml
persistentvolumeclaim/wrong-class created

List it:

kubectl get pvc wrong-class
NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
wrong-class   Pending                                      fast           <unset>                 5s

The PVC stays Pending. Its events show why:

kubectl describe pvc wrong-class
...
Events:
  Type     Reason              Age              From                         Message
  ----     ------              ----             ----                         -------
  Warning  ProvisioningFailed  3s (x2 over 6s)  persistentvolume-controller  storageclass.storage.k8s.io "fast" not found

Unlike data in Step 2, this PVC doesn’t wait for a Pod. Its StorageClass doesn’t exist. A Pod that used it would stay Pending as well.

Task 3: One for Each

Give each Pod of a StatefulSet its own PVC: copy ~/labs/2.5/redis.yaml to ~/labs/2.8/, set replicas: 2, add volumeClaimTemplates and mount the volume at /data. Each entry of volumeClaimTemplates is a PVC like the one in data-pvc.yaml (metadata and spec, without apiVersion and kind), and volumeMounts refers to it by its name. Add persistentVolumeClaimRetentionPolicy with whenDeleted: Delete, so that the Clean Up removes the PVCs as well. Then delete redis-1. Which PVC does the new redis-1 get?

You know all the commands you need.

Solution

Create the file redis.yaml in ~/labs/2.8:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  replicas: 2
  selector:
    matchLabels:
      app: redis
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:8.2
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

Create the StatefulSet:

kubectl apply -f ~/labs/2.8/redis.yaml

Wait until both Pods run (about 15 seconds) and list the PVCs:

kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data           Bound    pvc-f6a1add7-a5a1-4a04-a905-52dd2c299e45   1Gi        RWO            local-path     <unset>                 37s
data-redis-0   Bound    pvc-80c74fcc-08b7-49ab-893f-e696af46c21c   1Gi        RWO            local-path     <unset>                 16s
data-redis-1   Bound    pvc-79af92c5-d1c9-4004-998e-39767b564a21   1Gi        RWO            local-path     <unset>                 9s

Delete redis-1:

kubectl delete pod redis-1

Wait until the new Pod runs and list the PVCs again:

kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data           Bound    pvc-f6a1add7-a5a1-4a04-a905-52dd2c299e45   1Gi        RWO            local-path     <unset>                 54s
data-redis-0   Bound    pvc-80c74fcc-08b7-49ab-893f-e696af46c21c   1Gi        RWO            local-path     <unset>                 33s
data-redis-1   Bound    pvc-79af92c5-d1c9-4004-998e-39767b564a21   1Gi        RWO            local-path     <unset>                 26s

The StatefulSet created one PVC per Pod, named after the template and the Pod. The new redis-1 got its old PVC data-redis-1 back, together with its data. That’s the stable storage of a StatefulSet.

By default, the PVCs stay when you delete the StatefulSet, to protect the data. whenDeleted: Delete deletes them along with it, so the Clean Up removes them as well.


Clean Up

Delete all resources of this lab, since the following labs don’t need them:

kubectl delete -f ~/labs/2.8/
persistentvolumeclaim "data" deleted from default namespace
pod "keeper" deleted from default namespace
pod "scratch" deleted from default namespace

Kubernetes deletes the PV of the PVC data and its data along with it. If you did the optional tasks, their resources are deleted as well.