6. Challenges
These challenges take the message board further with what you learned in the labs. They are self-guided: there are no step-by-step instructions and no solutions. You decide how to solve each challenge, look things up in the labs and the documentation, and check the result yourself. A hint below each challenge tells you where to start.
| Rating | What to expect |
|---|---|
| Easy | Applies one lab to the message board |
| Medium | Combines several labs |
| Hard | Goes beyond the manifests, e.g. a change to the application image |
Starting point
All challenges build on the releases from chapter 4: the message board messages in the namespace messages-dev
and PostgreSQL postgresql in postgresql-dev. Challenge 1 also needs Traefik from 5.1, Challenge 2 the operator
from 5.2. Change the chart in ~/labs/4.2/messages and roll out every change with helm upgrade.
Packaging
Challenge 1: Ingress in the Chart
EasyAdd an Ingress to the chart, so that the message board is reachable at
http://messages.localhost:31337. Two values control it: ingress.enabled switches it on, ingress.host sets the
host name.
Where to look
- The Ingress from 5.1, Step 2
- Helm: Flow Control: with
if, Helm renders a part of a template only when a value is set
Challenge 2: PostgreSQL From CloudNativePG
MediumReplace the database from 4.1 with a PostgreSQL Cluster as in 5.2. Uninstall the release
postgresql first, then create the Cluster in the same namespace postgresql-dev and connect the backend to it.
The new database starts empty.
Where to look
- 5.2, Step 3: the Services that the operator creates for the Cluster
- The operator writes a generated password into the Secret
<cluster>-appinpostgresql-dev, but a Pod can only use the Secrets of its own namespace. Give the operator the user, password and database that your chart already uses:initdbin CloudNativePG: Bootstrap
Challenge 3: Migrations as a Job
HardThe backend creates its table itself when it starts. Move this into a Job that Helm runs before every install and upgrade, and remove it from the backend, so the table exists before the first backend Pod starts.
Where to look
- Helm: Chart Hooks:
pre-installandpre-upgrade - Hooks run before the other resources of the chart, including the Secret with the database password
- Build the changed backend image with a new tag and import it into your cluster, as in 3.1
Challenge 4: Two Environments With Kustomize
MediumRun the message board from 3.1 a second time, this time without Helm: one base with the manifests and two overlays
that deploy it into the namespaces kustomize-dev with one backend replica and kustomize-prod with three. No
manifest may be copied into an overlay. Compare the result with the two values files of 4.2, Task 2.
Where to look
- Declarative Management of Kubernetes Objects Using Kustomize:
Kustomize is part of
kubectl.kubectl kustomize <dir>shows the result,kubectl apply -k <dir>applies it - Kustomize only reads files in or below the directory of a
kustomization.yaml, so copy the manifests from~/labs/3.1into the base - The release from 4.2 already uses the NodePort 30088. Each overlay needs its own, e.g. 30086 and 30087
Operations
Challenge 5: Back Up and Restore
MediumBack up the database every night with a CronJob that writes a pg_dump into a
PersistentVolumeClaim. Then restore the backup into a new database and find your messages there.
Where to look
- The CronJob from 2.5 and the PersistentVolumeClaim from 2.8
- The image
postgres:18containspg_dumpandpsql, and both read the password from the variablePGPASSWORD kubectl create job <name> --from=cronjob/<cronjob>runs a CronJob right away- A Pod can only use the Secrets and PVCs of its own namespace
Challenge 6: Autoscale the Backend
MediumLet the backend scale between 2 and 6 Pods depending on its CPU usage, as part of the chart and switched on with a value. Generate load and watch the HorizontalPodAutoscaler add Pods, and remove them again a few minutes after the load stops.
Where to look
- 2.9, Step 5
- The autoscaler needs CPU requests on the backend
- The
replicasof the Deployment and the autoscaler fight over the number of Pods on everyhelm upgrade, so leavereplicasout of the template when the autoscaler is on - Load: a few Pods that call
http://backend.messages-dev:8000/messagesin a loop
Challenge 7: Survive a Node Drain
MediumEmpty a node with kubectl drain while you use the message board. It must stay reachable the whole time. Bring the
node back with kubectl uncordon.
Where to look
- Safely Drain a Node and Specifying a Disruption Budget
- The frontend runs a single replica. Spread more of them over the nodes as in 5.3
- The volumes of your cluster (
local-path) are bound to their node: a database Pod can’t move to another node
Challenge 8: Upgrade Without a Failed Request
HardCall http://localhost:30088/api/messages in a loop and count the failed requests while you restart the backend with
kubectl rollout restart. In your cluster, a few of about 800 requests fail. Change the chart until none do.
Where to look
- The readiness probe from 2.9 and the rolling update from 2.4 (
maxUnavailable,maxSurge) - Pod Lifecycle: Termination of Pods:
a Pod gets the stop signal while requests still reach it. A
preStophook can wait a moment curl -s -o /dev/null -w "%{http_code}\n" <url>prints only the status code
Security
Challenge 9: Lock Down the Database
MediumOnly the backend of the message board may connect to PostgreSQL, no other Pod in any namespace. Prove it with a client Pod that gets refused.
Where to look
- 5.4, Step 4 and Task 3:
podSelectorandnamespaceSelector - The NetworkPolicy belongs in the namespace of the database
- With CloudNativePG from Challenge 2, the operator and the instances must still reach each other, or the cluster falls apart
Challenge 10: Run Under the Restricted Standard
HardLabel the namespace messages-dev with pod-security.kubernetes.io/enforce: restricted and make the message board
run there.
Where to look
- Start with
warnas in 5.4, Task 2. The label only checks Pods when they are created, e.g. bykubectl rollout restart - The security context from 5.4, Step 2
- The frontend runs nginx as
rooton port 80. An unprivileged image such as nginxinc/nginx-unprivileged listens on port 8080, so the Service needs the newtargetPort - With a read-only file system, nginx needs
emptyDirvolumes where it writes
