Custom resource definitions (CRDs) add new resource types to the Kubernetes API, e.g. Cluster for a
PostgreSQL cluster and Backup for a backup. From now on, kubectl handles them like a Deployment or a Service.
A controller (the Pod in cnpg-system) watches the resources of these types and does what an experienced
database administrator would do: create instances, set up replication, take over when the primary fails.
grep cnpg keeps only the types of CloudNativePG, because other charts add their own, too.
Below NOTES, the chart shows an example cluster. Don’t apply it: you create your own in Step 2.
Step 2: Create a PostgreSQL Cluster
Create the directory of this lab and change into it:
mkdir -p ~/labs/5.2
cd ~/labs/5.2
Create the file database.yaml in ~/labs/5.2 with a Cluster of three instances:
The operator now sets up the instances one after another. This takes about two minutes. Check the status of the
cluster:
kubectl get cluster database
NAME AGE INSTANCES READY STATUS PRIMARY
database 104s 3 3 Cluster in healthy state database-1
If STATUS isn’t Cluster in healthy state yet, wait a moment and repeat the command. In the meantime, it shows
what the operator is doing, e.g. Setting up primary or Creating a new replica.
List the Pods:
kubectl get pods
NAME READY STATUS RESTARTS AGE
database-1 1/1 Running 0 84s
database-2 1/1 Running 0 54s
database-3 1/1 Running 0 14s
Explanation
Eight lines describe what you want: a PostgreSQL cluster with three instances and 1 GiB of storage each. The operator
worked out the rest. It started database-1 as the primary and created an empty database, then cloned
database-2 and database-3 from it as replicas. That’s why their AGE differs.
The operator manages the Pods and their PVCs directly. There’s no Deployment or StatefulSet behind them.
Step 3: Explore What the Operator Created
List the Pods with the label cnpg.io/instanceRole as an extra column:
kubectl get pods -L cnpg.io/instanceRole
NAME READY STATUS RESTARTS AGE INSTANCEROLE
database-1 1/1 Running 0 84s primary
database-2 1/1 Running 0 54s replica
database-3 1/1 Running 0 14s replica
List the Services and Secrets:
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
database-r ClusterIP 10.43.203.63 <none> 5432/TCP 106s
database-ro ClusterIP 10.43.54.248 <none> 5432/TCP 106s
database-rw ClusterIP 10.43.208.246 <none> 5432/TCP 106s
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 7h15m
kubectl get secrets
NAME TYPE DATA AGE
database-app kubernetes.io/basic-auth 11 106s
database-ca Opaque 2 106s
database-replication kubernetes.io/tls 2 106s
database-server kubernetes.io/tls 2 106s
Explanation
-L shows the value of a label as a column. The operator labels every Pod with its role and keeps the label up to
date.
The three Services select the Pods by this role:
Service
Leads to
database-rw
The primary, for reading and writing
database-ro
The replicas, for reading only
database-r
Any instance, for reading when any copy will do
database-app holds what an application needs: the user app, its password and ready-made connection strings for
the database app, e.g. uri and jdbc-uri. The other three Secrets hold the certificates the instances use to
talk to each other over TLS.
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
ERROR: cannot execute INSERT in a read-only transaction
command terminated with exit code 1
Explanation
psql app connects to the database app inside the container. The first line of each output only tells you that
kubectl exec picked the container postgres. The Pod also has an init container.
The row you wrote on the primary arrived on the replica within a second: PostgreSQL streams every change from the
primary to the replicas. The replicas only accept reads. Writes go to the primary, and applications reach it
through the Service database-rw.
Step 5: Delete the Primary
Watch the Pods and their roles in a second terminal:
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
text
----------------
Hello Operator
(1 row)
Explanation
Within about three seconds, the operator promoted a replica to the new primary and moved the label primary to it,
and with it the Service database-rw. While the old primary shut down, its label said unhealthy. Then the operator
recreated database-1 with its old PVC, this time as a replica that follows the new primary. After about 15 seconds,
the cluster was complete again, without losing the row.
This is the operating knowledge in code: which replica to promote, how to redirect the clients and how to turn the
old primary into a replica. With the StatefulSet from 3.1, Kubernetes would only restart the Pod, and the database
would be gone until it’s back.
Optional Tasks
Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need.
Open the solution if you get stuck.
Task 1: Connect Like an App
Read the key uri of the Secret database-app and connect with it from a temporary Pod with the image
ghcr.io/cloudnative-pg/postgresql:18.6-system-trixie. Do you land on the primary? Try the same through
database-ro.
Command
Effect
kubectl get secret <name> -o jsonpath="{.data.<key>}"
Shows a single value of the Secret, base64-encoded
psql "<uri>" -c "select pg_is_in_recovery()"
Connects with the URI. The answer t means it’s a replica
Solution
Read the connection string:
kubectl get secret database-app -o jsonpath="{.data.uri}"| base64 --decode
pg_is_in_recovery
-------------------
f
(1 row)
pod "psql" deleted from default namespace
Run it again with database-ro instead of database-rw in the connection string:
pg_is_in_recovery
-------------------
t
(1 row)
pod "psql" deleted from default namespace
f means you reached the primary, t a replica in recovery mode, which replays the changes of the primary. The
application only needs the Secret: the operator wrote the user, the password and the address of the right Service into
it.
Task 2: Scale Out
Grow the cluster to four instances. Which role does the new instance get? Shrink it back to three afterwards.
You know all the commands you need.
Solution
Set instances: 4 in database.yaml and apply it:
kubectl apply -f ~/labs/5.2/database.yaml
cluster.postgresql.cnpg.io/database configured
After about 20 seconds, the new instance is ready:
kubectl get pods -L cnpg.io/instanceRole
NAME READY STATUS RESTARTS AGE INSTANCEROLE
database-1 1/1 Running 0 42s replica
database-2 1/1 Running 0 102s primary
database-3 1/1 Running 0 62s replica
database-4 1/1 Running 0 11s replica
If database-4 isn’t 1/1 yet, wait a moment and repeat the command. Until then, the list also shows the Pod
database-4-join-…: it cloned the data from the primary, and the operator deletes it once the new instance runs.
The operator added the new instance as one more replica. Set instances: 3 again and apply the file: the operator
removes database-4 together with its PVC within seconds.
Task 3: Operator on Strike
Stop the operator by scaling its Deployment cnpg-cloudnative-pg in the namespace cnpg-system to zero. Delete a
replica. What happens? Then start the operator again.
Within about 15 seconds, database-3 is back as a replica. The database itself kept running all the time: the
operator only acts when something changes: it compares the desired state in the Cluster with the actual state
and closes the gap.
Clean Up
Delete the PostgreSQL cluster:
kubectl delete -f ~/labs/5.2/
cluster.postgresql.cnpg.io "database" deleted from default namespace
The Pods, PVCs, Services and Secrets of the cluster belong to the Cluster resource, so Kubernetes deletes them,
too. The operator keeps running for the challenges in chapter 6.