6. Challenges

Challenges Challenges

These challenges take the message board further with what you learned in the labs. They are self-guided: there are no step-by-step instructions and no solutions. You decide how to solve each challenge, look things up in the labs and the documentation, and check the result yourself. A hint below each challenge tells you where to start.

RatingWhat to expect
EasyApplies one lab to the message board
MediumCombines several labs
HardGoes beyond the manifests, e.g. a change to the application image
Starting point

All challenges build on the releases from chapter 4: the message board messages in the namespace messages-dev and PostgreSQL postgresql in postgresql-dev. Challenge 1 also needs Traefik from 5.1, Challenge 2 the operator from 5.2. Change the chart in ~/labs/4.2/messages and roll out every change with helm upgrade.

Packaging

Challenge 1: Ingress in the Chart

Easy

Add an Ingress to the chart, so that the message board is reachable at http://messages.localhost:31337. Two values control it: ingress.enabled switches it on, ingress.host sets the host name.

Where to look
  • The Ingress from 5.1, Step 2
  • Helm: Flow Control: with if, Helm renders a part of a template only when a value is set

Challenge 2: PostgreSQL From CloudNativePG

Medium

Replace the database from 4.1 with a PostgreSQL Cluster as in 5.2. Uninstall the release postgresql first, then create the Cluster in the same namespace postgresql-dev and connect the backend to it. The new database starts empty.

Where to look
  • 5.2, Step 3: the Services that the operator creates for the Cluster
  • The operator writes a generated password into the Secret <cluster>-app in postgresql-dev, but a Pod can only use the Secrets of its own namespace. Give the operator the user, password and database that your chart already uses: initdb in CloudNativePG: Bootstrap

Challenge 3: Migrations as a Job

Hard

The backend creates its table itself when it starts. Move this into a Job that Helm runs before every install and upgrade, and remove it from the backend, so the table exists before the first backend Pod starts.

Where to look
  • Helm: Chart Hooks: pre-install and pre-upgrade
  • Hooks run before the other resources of the chart, including the Secret with the database password
  • Build the changed backend image with a new tag and import it into your cluster, as in 3.1

Challenge 4: Two Environments With Kustomize

Medium

Run the message board from 3.1 a second time, this time without Helm: one base with the manifests and two overlays that deploy it into the namespaces kustomize-dev with one backend replica and kustomize-prod with three. No manifest may be copied into an overlay. Compare the result with the two values files of 4.2, Task 2.

Where to look
  • Declarative Management of Kubernetes Objects Using Kustomize: Kustomize is part of kubectl. kubectl kustomize <dir> shows the result, kubectl apply -k <dir> applies it
  • Kustomize only reads files in or below the directory of a kustomization.yaml, so copy the manifests from ~/labs/3.1 into the base
  • The release from 4.2 already uses the NodePort 30088. Each overlay needs its own, e.g. 30086 and 30087

Operations

Challenge 5: Back Up and Restore

Medium

Back up the database every night with a CronJob that writes a pg_dump into a PersistentVolumeClaim. Then restore the backup into a new database and find your messages there.

Where to look
  • The CronJob from 2.5 and the PersistentVolumeClaim from 2.8
  • The image postgres:18 contains pg_dump and psql, and both read the password from the variable PGPASSWORD
  • kubectl create job <name> --from=cronjob/<cronjob> runs a CronJob right away
  • A Pod can only use the Secrets and PVCs of its own namespace

Challenge 6: Autoscale the Backend

Medium

Let the backend scale between 2 and 6 Pods depending on its CPU usage, as part of the chart and switched on with a value. Generate load and watch the HorizontalPodAutoscaler add Pods, and remove them again a few minutes after the load stops.

Where to look
  • 2.9, Step 5
  • The autoscaler needs CPU requests on the backend
  • The replicas of the Deployment and the autoscaler fight over the number of Pods on every helm upgrade, so leave replicas out of the template when the autoscaler is on
  • Load: a few Pods that call http://backend.messages-dev:8000/messages in a loop

Challenge 7: Survive a Node Drain

Medium

Empty a node with kubectl drain while you use the message board. It must stay reachable the whole time. Bring the node back with kubectl uncordon.

Where to look

Challenge 8: Upgrade Without a Failed Request

Hard

Call http://localhost:30088/api/messages in a loop and count the failed requests while you restart the backend with kubectl rollout restart. In your cluster, a few of about 800 requests fail. Change the chart until none do.

Where to look
  • The readiness probe from 2.9 and the rolling update from 2.4 (maxUnavailable, maxSurge)
  • Pod Lifecycle: Termination of Pods: a Pod gets the stop signal while requests still reach it. A preStop hook can wait a moment
  • curl -s -o /dev/null -w "%{http_code}\n" <url> prints only the status code

Security

Challenge 9: Lock Down the Database

Medium

Only the backend of the message board may connect to PostgreSQL, no other Pod in any namespace. Prove it with a client Pod that gets refused.

Where to look
  • 5.4, Step 4 and Task 3: podSelector and namespaceSelector
  • The NetworkPolicy belongs in the namespace of the database
  • With CloudNativePG from Challenge 2, the operator and the instances must still reach each other, or the cluster falls apart

Challenge 10: Run Under the Restricted Standard

Hard

Label the namespace messages-dev with pod-security.kubernetes.io/enforce: restricted and make the message board run there.

Where to look
  • Start with warn as in 5.4, Task 2. The label only checks Pods when they are created, e.g. by kubectl rollout restart
  • The security context from 5.4, Step 2
  • The frontend runs nginx as root on port 80. An unprivileged image such as nginxinc/nginx-unprivileged listens on port 8080, so the Service needs the new targetPort
  • With a read-only file system, nginx needs emptyDir volumes where it writes