5.4. Security

In this lab, you restrict what a workload may do: which API calls it may make, what its container may do, which Pods a namespace admits and who may talk to a Pod.

Step 1: Grant a ServiceAccount Access With RBAC

Create the directory of this lab and change into it:

mkdir -p ~/labs/5.4
cd ~/labs/5.4

Check whether the ServiceAccount reader, which doesn’t exist yet, may list Pods:

kubectl auth can-i list pods --as=system:serviceaccount:default:reader
no

Create the file reader.yaml in ~/labs/5.4 with the ServiceAccount, a Role that allows reading Pods and a RoleBinding that grants the Role to the ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: reader
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: reader-pod-reader
subjects:
  - kind: ServiceAccount
    name: reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader

Apply it:

kubectl apply -f reader.yaml
serviceaccount/reader created
role.rbac.authorization.k8s.io/pod-reader created
rolebinding.rbac.authorization.k8s.io/reader-pod-reader created

Check again what the ServiceAccount may do:

kubectl auth can-i list pods --as=system:serviceaccount:default:reader
yes
kubectl auth can-i delete pods --as=system:serviceaccount:default:reader
no
kubectl auth can-i list secrets --as=system:serviceaccount:default:reader
no

Explanation

A ServiceAccount is the identity of a workload: a Pod talks to the API server in its name. Its user name is system:serviceaccount:<namespace>:<name>, and --as lets you ask the API server in the name of this user.

RBAC allows only what a rule permits. Everything else is forbidden:

ResourceDefines
RoleWhat is allowed: the verbs on the resources, e.g. list on pods
RoleBindingWho gets it: the subjects, here the ServiceAccount reader

Both are valid only in their namespace, default. So reader may read Pods there, but neither delete them nor read Secrets. apiGroups: [""] stands for the core group: Pods have no group, only v1, as kubectl api-resources showed in 2.2.


Step 2: Run a Container With a Restrictive Security Context

Check which user a container runs as by default:

kubectl run plain --image=busybox:1.37 --rm -it --restart=Never -- id
uid=0(root) gid=0(root) groups=0(root),10(wheel)
pod "plain" deleted from default namespace

Create the file secure.yaml in ~/labs/5.4 with a Pod that restricts its container:

apiVersion: v1
kind: Pod
metadata:
  name: secure
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh", "-c", "trap 'exit 0' TERM; sleep 3600 & wait"]
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]

Apply it and list the Pods:

kubectl apply -f secure.yaml
pod/secure created
kubectl get pods
NAME     READY   STATUS    RESTARTS   AGE
secure   1/1     Running   0          1s

If the Pod isn’t Running yet, wait a moment and repeat the command. Check the user and try to write a file:

kubectl exec secure -- id
uid=1000 gid=1000 groups=1000
kubectl exec secure -- touch /tmp/test
touch: /tmp/test: Read-only file system
command terminated with exit code 1

Explanation

Unless told otherwise, the busybox container runs as root. The security context takes rights away, either at the level of the Pod for all containers or for a single container:

SettingEffect
runAsNonRoot: trueRefuses to start a container as root
runAsUser, runAsGroupRuns the processes as user and group 1000
seccompProfile: RuntimeDefaultBlocks system calls that containers rarely need
allowPrivilegeEscalation: falseA process can’t gain more rights than it started with, e.g. through sudo
readOnlyRootFilesystem: trueThe container can’t change its file system
capabilities.drop: ["ALL"]Removes the special powers that Linux gives root

If an attacker takes over the application, they get only these limited rights. An application that needs to write files, e.g. to /tmp, gets an emptyDir volume there, as in 2.8.


Step 3: Enforce Pod Security Standards on a Namespace

Create the file restricted.yaml in ~/labs/5.4 with a namespace that admits only Pods meeting the standard restricted:

apiVersion: v1
kind: Namespace
metadata:
  name: restricted
  labels:
    pod-security.kubernetes.io/enforce: restricted

Apply it and start the busybox Pod from Step 2 in it:

kubectl apply -f restricted.yaml
namespace/restricted created
kubectl run plain --image=busybox:1.37 -n restricted --rm -it --restart=Never -- id
Error from server (Forbidden): pods "plain" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "plain" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "plain" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "plain" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "plain" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Apply the Pod secure in the same namespace:

kubectl apply -f secure.yaml -n restricted
pod/secure created
kubectl get pods -n restricted
NAME     READY   STATUS    RESTARTS   AGE
secure   1/1     Running   0          2s

If the Pod isn’t Running yet, wait a moment and repeat the command.

Explanation

The label on the namespace switches on the check: the API server compares every new Pod with the Pod Security Standard restricted and rejects it if a setting is missing. The error lists each of them. These are exactly the settings of the Pod secure from Step 2, which is why it was admitted.

There are three standards: privileged allows everything, baseline blocks the most dangerous settings, and restricted demands the settings of Step 2.


Step 4: Isolate Pods With a NetworkPolicy

Create the file web.yaml in ~/labs/5.4 with a web server and its Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.29
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

Apply it and call the web server from a temporary Pod:

kubectl apply -f web.yaml
deployment.apps/web created
service/web created
kubectl run client --image=busybox:1.37 --rm -it --restart=Never -- sh -c 'wget -q -O - -T 3 http://web | grep title'
<title>Welcome to nginx!</title>
pod "client" deleted from default namespace

As in 2.6, kubectl may also print a few more lines, e.g. If you don't see a command prompt, try pressing enter. or Session ended, resume using 'kubectl attach …' command. You can ignore them.

Create the file web-policy.yaml in ~/labs/5.4 with a NetworkPolicy that lets only Pods with the label access: "true" reach the web server:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
    - from:
        - podSelector:
            matchLabels:
              access: "true"

Apply it and call the web server again, once without and once with the label:

kubectl apply -f web-policy.yaml
networkpolicy.networking.k8s.io/web created
kubectl run client --image=busybox:1.37 --rm -it --restart=Never -- sh -c 'wget -q -O - -T 3 http://web | grep title'
wget: can't connect to remote host (10.43.37.199): Connection refused
pod "client" deleted from default namespace
pod default/client terminated (Error)
kubectl run client --labels=access=true --image=busybox:1.37 --rm -it --restart=Never -- sh -c 'wget -q -O - -T 3 http://web | grep title'
<title>Welcome to nginx!</title>
pod "client" deleted from default namespace

Explanation

Without a NetworkPolicy, every Pod can reach every other Pod. As soon as a policy selects a Pod (here with podSelector all Pods with app: web), only the traffic the policy allows gets in: connections from Pods with the label access: "true" in the same namespace. --labels gives the temporary Pod this label. In the manifest, the quotes make true a string: label values are always strings.

In your cluster, k3s enforces the policy with its built-in network policy controller and refuses the connection right away. wget fails, so the temporary Pod ends with Error. Other network plugins may drop the connection instead, so the client runs into its timeout of three seconds (-T 3).


Optional Tasks

Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need. Open the solution if you get stuck.

Task 1: Inside Job

Run a Pod inside with the ServiceAccount reader and the image registry.k8s.io/kubectl:v1.35.5, whose container runs kubectl get pods,secrets. What does its log show?

The field serviceAccountName in the Pod spec sets the ServiceAccount. The image starts kubectl with the args of the container, e.g. args: ["get", "pods,secrets"]. You know all the commands you need.

Solution

Create the file inside.yaml in ~/labs/5.4:

apiVersion: v1
kind: Pod
metadata:
  name: inside
spec:
  serviceAccountName: reader
  restartPolicy: Never
  containers:
    - name: kubectl
      image: registry.k8s.io/kubectl:v1.35.5
      args: ["get", "pods,secrets"]

Apply it, wait about 10 seconds and read the log:

kubectl apply -f ~/labs/5.4/inside.yaml
pod/inside created
kubectl logs inside
NAME                  READY   STATUS              RESTARTS   AGE
inside                0/1     ContainerCreating   0          2s
secure                1/1     Running             0          18s
web-f9b8bfb4c-gnbk6   1/1     Running             0          15s
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:reader" cannot list resource "secrets" in API group "" in the namespace "default"

kubectl inside the Pod used the token of the ServiceAccount, which Kubernetes mounts into every Pod. The API server applied the Role from Step 1: the Pods were listed, the Secrets were forbidden. That’s why the Pod ended with the status Error.

Task 2: Warning Shot

Create a namespace warned that only warns about Pods that violate the standard restricted, instead of rejecting them. Start the busybox Pod from Step 2 in it.

The label key is pod-security.kubernetes.io/warn. You know all the commands you need.

Solution

Create the file warned.yaml in ~/labs/5.4:

apiVersion: v1
kind: Namespace
metadata:
  name: warned
  labels:
    pod-security.kubernetes.io/warn: restricted
kubectl apply -f ~/labs/5.4/warned.yaml
namespace/warned created
kubectl run plain --image=busybox:1.37 -n warned --rm -it --restart=Never -- id
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "plain" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "plain" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "plain" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "plain" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
uid=0(root) gid=0(root) groups=0(root),10(wheel)
pod "plain" deleted from warned namespace

The Pod ran as root, but you got the same list as in Step 3 as a warning. That’s how you introduce a standard: first warn, fix the workloads that show up, then enforce.

Task 3: Wrong Neighborhood

Create a namespace other and call the web server from a Pod with the label access=true in it. Why is the connection refused? Allow it with a second NetworkPolicy, without changing the first one.

A Pod in another namespace reaches the Service as web.default. In the from part of a policy, a namespaceSelector selects namespaces. Every namespace carries the label kubernetes.io/metadata.name with its name. You know all the commands you need.

Solution

Create the file other.yaml in ~/labs/5.4 and apply it:

apiVersion: v1
kind: Namespace
metadata:
  name: other
kubectl apply -f ~/labs/5.4/other.yaml
namespace/other created
kubectl run client -n other --labels=access=true --image=busybox:1.37 --rm -it --restart=Never -- sh -c 'wget -q -O - -T 3 http://web.default | grep title'
wget: can't connect to remote host (10.43.37.199): Connection refused
pod "client" deleted from other namespace
pod other/client terminated (Error)

The podSelector in from only matches Pods in the namespace of the policy. Create the file web-policy-other.yaml in ~/labs/5.4:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-from-other
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: other
          podSelector:
            matchLabels:
              access: "true"
kubectl apply -f ~/labs/5.4/web-policy-other.yaml
networkpolicy.networking.k8s.io/web-from-other created

Run the client in other again:

kubectl run client -n other --labels=access=true --image=busybox:1.37 --rm -it --restart=Never -- sh -c 'wget -q -O - -T 3 http://web.default | grep title'
<title>Welcome to nginx!</title>
pod "client" deleted from other namespace

namespaceSelector and podSelector in the same entry must both match: Pods with access: "true" in the namespace other. NetworkPolicies add up: the web server now accepts what either policy allows.


Clean Up

Delete all resources of this lab, including the namespaces:

kubectl delete -f ~/labs/5.4/
serviceaccount "reader" deleted from default namespace
role.rbac.authorization.k8s.io "pod-reader" deleted from default namespace
rolebinding.rbac.authorization.k8s.io "reader-pod-reader" deleted from default namespace
namespace "restricted" deleted
pod "secure" deleted from default namespace
networkpolicy.networking.k8s.io "web" deleted from default namespace
deployment.apps "web" deleted from default namespace
service "web" deleted from default namespace

It takes about ten seconds, because deleting a namespace waits for its Pods. If you did the optional tasks, the output also lists inside, other, warned and web-from-other.


Complete files

~/labs/5.4/reader.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: reader
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: reader-pod-reader
subjects:
  - kind: ServiceAccount
    name: reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader

~/labs/5.4/secure.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: secure
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh", "-c", "trap 'exit 0' TERM; sleep 3600 & wait"]
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]

~/labs/5.4/restricted.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: restricted
  labels:
    pod-security.kubernetes.io/enforce: restricted

~/labs/5.4/web.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.29
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

~/labs/5.4/web-policy.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
    - from:
        - podSelector:
            matchLabels:
              access: "true"