5.2. Operators

In this lab, you install the operator CloudNativePG and let it run a replicated PostgreSQL cluster, including the failover when the primary goes down.

Step 1: Install the Operator

Install the chart of CloudNativePG as the release cnpg into the new namespace cnpg-system:

helm install cnpg oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg --version 0.29.1 \
  -n cnpg-system --create-namespace
Pulled: ghcr.io/cloudnative-pg/charts/cloudnative-pg:0.29.1
Digest: sha256:3245fa051bb21d0dd9246272f57a2760b4234817a164a3167d2a1e7fb9f63fd9
NAME: cnpg
LAST DEPLOYED: Tue Oct 13 17:23:39 2026
NAMESPACE: cnpg-system
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
TEST SUITE: None
NOTES:
CloudNativePG operator should be installed in namespace "cnpg-system".
...

Wait about 10 seconds and list the Pod of the operator:

kubectl get pods -n cnpg-system
NAME                                   READY   STATUS    RESTARTS   AGE
cnpg-cloudnative-pg-84cd4568f9-ppp7d   1/1     Running   0          6s

If the Pod isn’t 1/1 yet, wait a moment and repeat the command. List the resource types that the chart added:

kubectl get crds | grep cnpg
backups.postgresql.cnpg.io                2026-10-13T17:23:40Z
clusterimagecatalogs.postgresql.cnpg.io   2026-10-13T17:23:40Z
clusters.postgresql.cnpg.io               2026-10-13T17:23:40Z
databaseroles.postgresql.cnpg.io          2026-10-13T17:23:40Z
databases.postgresql.cnpg.io              2026-10-13T17:23:40Z
failoverquorums.postgresql.cnpg.io        2026-10-13T17:23:40Z
imagecatalogs.postgresql.cnpg.io          2026-10-13T17:23:40Z
poolers.postgresql.cnpg.io                2026-10-13T17:23:40Z
publications.postgresql.cnpg.io           2026-10-13T17:23:40Z
scheduledbackups.postgresql.cnpg.io       2026-10-13T17:23:40Z
subscriptions.postgresql.cnpg.io          2026-10-13T17:23:40Z

Explanation

An operator consists of two parts:

  • Custom resource definitions (CRDs) add new resource types to the Kubernetes API, e.g. Cluster for a PostgreSQL cluster and Backup for a backup. From now on, kubectl handles them like a Deployment or a Service.
  • A controller (the Pod in cnpg-system) watches the resources of these types and does what an experienced database administrator would do: create instances, set up replication, take over when the primary fails.

grep cnpg keeps only the types of CloudNativePG, because other charts add their own, too.

Below NOTES, the chart shows an example cluster. Don’t apply it: you create your own in Step 2.


Step 2: Create a PostgreSQL Cluster

Create the directory of this lab and change into it:

mkdir -p ~/labs/5.2
cd ~/labs/5.2

Create the file database.yaml in ~/labs/5.2 with a Cluster of three instances:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: database
spec:
  instances: 3
  storage:
    size: 1Gi

Apply it:

kubectl apply -f database.yaml
cluster.postgresql.cnpg.io/database created

The operator now sets up the instances one after another. This takes about two minutes. Check the status of the cluster:

kubectl get cluster database
NAME       AGE    INSTANCES   READY   STATUS                     PRIMARY
database   104s   3           3       Cluster in healthy state   database-1

If STATUS isn’t Cluster in healthy state yet, wait a moment and repeat the command. In the meantime, it shows what the operator is doing, e.g. Setting up primary or Creating a new replica.

List the Pods:

kubectl get pods
NAME         READY   STATUS    RESTARTS   AGE
database-1   1/1     Running   0          84s
database-2   1/1     Running   0          54s
database-3   1/1     Running   0          14s

Explanation

Eight lines describe what you want: a PostgreSQL cluster with three instances and 1 GiB of storage each. The operator worked out the rest. It started database-1 as the primary and created an empty database, then cloned database-2 and database-3 from it as replicas. That’s why their AGE differs.

The operator manages the Pods and their PVCs directly. There’s no Deployment or StatefulSet behind them.


Step 3: Explore What the Operator Created

List the Pods with the label cnpg.io/instanceRole as an extra column:

kubectl get pods -L cnpg.io/instanceRole
NAME         READY   STATUS    RESTARTS   AGE   INSTANCEROLE
database-1   1/1     Running   0          84s   primary
database-2   1/1     Running   0          54s   replica
database-3   1/1     Running   0          14s   replica

List the Services and Secrets:

kubectl get services
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
database-r    ClusterIP   10.43.203.63    <none>        5432/TCP   106s
database-ro   ClusterIP   10.43.54.248    <none>        5432/TCP   106s
database-rw   ClusterIP   10.43.208.246   <none>        5432/TCP   106s
kubernetes    ClusterIP   10.43.0.1       <none>        443/TCP    7h15m
kubectl get secrets
NAME                   TYPE                       DATA   AGE
database-app           kubernetes.io/basic-auth   11     106s
database-ca            Opaque                     2      106s
database-replication   kubernetes.io/tls          2      106s
database-server        kubernetes.io/tls          2      106s

Explanation

-L shows the value of a label as a column. The operator labels every Pod with its role and keeps the label up to date.

The three Services select the Pods by this role:

ServiceLeads to
database-rwThe primary, for reading and writing
database-roThe replicas, for reading only
database-rAny instance, for reading when any copy will do

database-app holds what an application needs: the user app, its password and ready-made connection strings for the database app, e.g. uri and jdbc-uri. The other three Secrets hold the certificates the instances use to talk to each other over TLS.


Step 4: Write on the Primary, Read on a Replica

Create a table on the primary and insert a row:

kubectl exec database-1 -- psql app -c "create table notes (text text)" -c "insert into notes values ('Hello Operator')"
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
CREATE TABLE
INSERT 0 1

Read the table on a replica:

kubectl exec database-2 -- psql app -c "select * from notes"
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
      text
----------------
 Hello Operator
(1 row)

Try to insert a row on the replica:

kubectl exec database-2 -- psql app -c "insert into notes values ('Hello Replica')"
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
ERROR:  cannot execute INSERT in a read-only transaction
command terminated with exit code 1

Explanation

psql app connects to the database app inside the container. The first line of each output only tells you that kubectl exec picked the container postgres. The Pod also has an init container.

The row you wrote on the primary arrived on the replica within a second: PostgreSQL streams every change from the primary to the replicas. The replicas only accept reads. Writes go to the primary, and applications reach it through the Service database-rw.


Step 5: Delete the Primary

Watch the Pods and their roles in a second terminal:

kubectl get pods -L cnpg.io/instanceRole --watch

Delete the primary in the first terminal:

kubectl delete pod database-1
pod "database-1" deleted from default namespace

The watch shows the failover:

NAME         READY   STATUS    RESTARTS   AGE   INSTANCEROLE
database-1   1/1     Running   0          85s   primary
database-2   1/1     Running   0          55s   replica
database-3   1/1     Running   0          15s   replica
database-1   1/1     Terminating   0          87s   primary
database-1   1/1     Terminating   0          87s   primary
database-1   1/1     Terminating   0          88s   unhealthy
database-1   0/1     Completed     0          88s   unhealthy
database-1   0/1     Completed     0          89s   unhealthy
database-1   0/1     Completed     0          89s   unhealthy
database-2   1/1     Running       0          60s   primary
database-1   0/1     Pending       0          0s
database-1   0/1     Pending       0          0s
database-1   0/1     Init:0/1      0          0s
database-1   0/1     Init:0/1      0          1s
database-1   0/1     PodInitializing   0          2s
database-1   0/1     Running           0          3s
database-1   0/1     Running           0          4s    replica
database-1   0/1     Running           0          10s   replica
database-1   1/1     Running           0          11s   replica

Stop the watch with Ctrl+C and check the cluster:

kubectl get cluster database
NAME       AGE    INSTANCES   READY   STATUS                     PRIMARY
database   2m7s   3           3       Cluster in healthy state   database-2

Read the table on the new primary. If the operator picked database-3 in your cluster, use that one:

kubectl exec database-2 -- psql app -c "select * from notes"
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
      text
----------------
 Hello Operator
(1 row)

Explanation

Within about three seconds, the operator promoted a replica to the new primary and moved the label primary to it, and with it the Service database-rw. While the old primary shut down, its label said unhealthy. Then the operator recreated database-1 with its old PVC, this time as a replica that follows the new primary. After about 15 seconds, the cluster was complete again, without losing the row.

This is the operating knowledge in code: which replica to promote, how to redirect the clients and how to turn the old primary into a replica. With the StatefulSet from 3.1, Kubernetes would only restart the Pod, and the database would be gone until it’s back.


Optional Tasks

Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need. Open the solution if you get stuck.

Task 1: Connect Like an App

Read the key uri of the Secret database-app and connect with it from a temporary Pod with the image ghcr.io/cloudnative-pg/postgresql:18.6-system-trixie. Do you land on the primary? Try the same through database-ro.

CommandEffect
kubectl get secret <name> -o jsonpath="{.data.<key>}"Shows a single value of the Secret, base64-encoded
psql "<uri>" -c "select pg_is_in_recovery()"Connects with the URI. The answer t means it’s a replica
Solution

Read the connection string:

kubectl get secret database-app -o jsonpath="{.data.uri}" | base64 --decode
postgresql://app:FtXwVlEsyqV37hHEAZXcaHLkmP0VijbY42kbPjO58zpddgWVkGvt5uIl2lYNdq2O@database-rw.default:5432/app

Your password differs. Connect with it from a temporary Pod. Replace <uri> with your connection string:

kubectl run psql --image=ghcr.io/cloudnative-pg/postgresql:18.6-system-trixie --rm -it --restart=Never -- \
  psql "<uri>" -c "select pg_is_in_recovery()"
 pg_is_in_recovery
-------------------
 f
(1 row)

pod "psql" deleted from default namespace

Run it again with database-ro instead of database-rw in the connection string:

 pg_is_in_recovery
-------------------
 t
(1 row)

pod "psql" deleted from default namespace

f means you reached the primary, t a replica in recovery mode, which replays the changes of the primary. The application only needs the Secret: the operator wrote the user, the password and the address of the right Service into it.

Task 2: Scale Out

Grow the cluster to four instances. Which role does the new instance get? Shrink it back to three afterwards.

You know all the commands you need.

Solution

Set instances: 4 in database.yaml and apply it:

kubectl apply -f ~/labs/5.2/database.yaml
cluster.postgresql.cnpg.io/database configured

After about 20 seconds, the new instance is ready:

kubectl get pods -L cnpg.io/instanceRole
NAME         READY   STATUS    RESTARTS   AGE    INSTANCEROLE
database-1   1/1     Running   0          42s    replica
database-2   1/1     Running   0          102s   primary
database-3   1/1     Running   0          62s    replica
database-4   1/1     Running   0          11s    replica

If database-4 isn’t 1/1 yet, wait a moment and repeat the command. Until then, the list also shows the Pod database-4-join-…: it cloned the data from the primary, and the operator deletes it once the new instance runs.

The operator added the new instance as one more replica. Set instances: 3 again and apply the file: the operator removes database-4 together with its PVC within seconds.

Task 3: Operator on Strike

Stop the operator by scaling its Deployment cnpg-cloudnative-pg in the namespace cnpg-system to zero. Delete a replica. What happens? Then start the operator again.

CommandEffect
kubectl scale deployment <name> -n <namespace> --replicas=<number>Sets the number of replicas of a Deployment

kubectl get pods -L is known from Step 3.

Solution

Stop the operator and delete a replica. Check the roles first and pick one that is replica:

kubectl scale deployment cnpg-cloudnative-pg -n cnpg-system --replicas=0
deployment.apps/cnpg-cloudnative-pg scaled
kubectl delete pod database-3
pod "database-3" deleted from default namespace

Wait 30 seconds and check the Pods and the cluster:

kubectl get pods -L cnpg.io/instanceRole
NAME         READY   STATUS    RESTARTS   AGE     INSTANCEROLE
database-1   1/1     Running   0          83s     replica
database-2   1/1     Running   0          2m23s   primary
kubectl get cluster database
NAME       AGE     INSTANCES   READY   STATUS                     PRIMARY
database   3m15s   3           3       Cluster in healthy state   database-2

Nobody recreates the Pod, and the status still claims three ready instances. The operator that keeps it up to date isn’t running. Start it again:

kubectl scale deployment cnpg-cloudnative-pg -n cnpg-system --replicas=1
deployment.apps/cnpg-cloudnative-pg scaled

Within about 15 seconds, database-3 is back as a replica. The database itself kept running all the time: the operator only acts when something changes: it compares the desired state in the Cluster with the actual state and closes the gap.


Clean Up

Delete the PostgreSQL cluster:

kubectl delete -f ~/labs/5.2/
cluster.postgresql.cnpg.io "database" deleted from default namespace

The Pods, PVCs, Services and Secrets of the cluster belong to the Cluster resource, so Kubernetes deletes them, too. The operator keeps running for the challenges in chapter 6.


Complete files

~/labs/5.2/database.yaml:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: database
spec:
  instances: 3
  storage:
    size: 1Gi