In this lab, you restrict what a workload may do: which API calls it may make, what its container may do, which Pods
a namespace admits and who may talk to a Pod.
Step 1: Grant a ServiceAccount Access With RBAC
Create the directory of this lab and change into it:
mkdir -p ~/labs/5.4
cd ~/labs/5.4
Check whether the ServiceAccount reader, which doesn’t exist yet, may list Pods:
kubectl auth can-i list pods --as=system:serviceaccount:default:reader
no
Create the file reader.yaml in ~/labs/5.4 with the ServiceAccount, a
Role that allows reading Pods and a RoleBinding that grants the Role to the
ServiceAccount:
kubectl auth can-i list secrets --as=system:serviceaccount:default:reader
no
Explanation
A ServiceAccount is the identity of a workload: a Pod talks to the API server in its name. Its user name is
system:serviceaccount:<namespace>:<name>, and --as lets you ask the API server in the name of this user.
RBAC allows only what a rule permits. Everything else is forbidden:
Resource
Defines
Role
What is allowed: the verbs on the resources, e.g. list on pods
RoleBinding
Who gets it: the subjects, here the ServiceAccount reader
Both are valid only in their namespace, default. So reader may read Pods there, but neither delete them nor
read Secrets. apiGroups: [""] stands for the core group: Pods have no group, only v1, as kubectl api-resources
showed in 2.2.
Step 2: Run a Container With a Restrictive Security Context
Check which user a container runs as by default:
kubectl run plain --image=busybox:1.37 --rm -it --restart=Never -- id
uid=0(root) gid=0(root) groups=0(root),10(wheel)
pod "plain" deleted from default namespace
Create the file secure.yaml in ~/labs/5.4 with a Pod that restricts its container:
NAME READY STATUS RESTARTS AGE
secure 1/1 Running 0 1s
If the Pod isn’t Running yet, wait a moment and repeat the command. Check the user and try to write a file:
kubectl exec secure -- id
uid=1000 gid=1000 groups=1000
kubectl exec secure -- touch /tmp/test
touch: /tmp/test: Read-only file system
command terminated with exit code 1
Explanation
Unless told otherwise, the busybox container runs as root. The security context takes rights away, either at the level
of the Pod for all containers or for a single container:
Setting
Effect
runAsNonRoot: true
Refuses to start a container as root
runAsUser, runAsGroup
Runs the processes as user and group 1000
seccompProfile: RuntimeDefault
Blocks system calls that containers rarely need
allowPrivilegeEscalation: false
A process can’t gain more rights than it started with, e.g. through sudo
readOnlyRootFilesystem: true
The container can’t change its file system
capabilities.drop: ["ALL"]
Removes the special powers that Linux gives root
If an attacker takes over the application, they get only these limited rights. An application that needs to write
files, e.g. to /tmp, gets an emptyDir volume there, as in 2.8.
Step 3: Enforce Pod Security Standards on a Namespace
Create the file restricted.yaml in ~/labs/5.4 with a namespace that admits only Pods meeting the
standard restricted:
Apply it and start the busybox Pod from Step 2 in it:
kubectl apply -f restricted.yaml
namespace/restricted created
kubectl run plain --image=busybox:1.37 -n restricted --rm -it --restart=Never -- id
Error from server (Forbidden): pods "plain" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "plain" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "plain" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "plain" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "plain" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Apply the Pod secure in the same namespace:
kubectl apply -f secure.yaml -n restricted
pod/secure created
kubectl get pods -n restricted
NAME READY STATUS RESTARTS AGE
secure 1/1 Running 0 2s
If the Pod isn’t Running yet, wait a moment and repeat the command.
Explanation
The label on the namespace switches on the check: the API server compares every new Pod with the Pod Security
Standard restricted and rejects it if a setting is missing. The error lists each of them. These are exactly the
settings of the Pod secure from Step 2, which is why it was admitted.
There are three standards: privileged allows everything, baseline blocks the most dangerous settings, and
restricted demands the settings of Step 2.
Step 4: Isolate Pods With a NetworkPolicy
Create the file web.yaml in ~/labs/5.4 with a web server and its Service:
Apply it and call the web server from a temporary Pod:
kubectl apply -f web.yaml
deployment.apps/web created
service/web created
kubectl run client --image=busybox:1.37 --rm -it --restart=Never -- sh -c 'wget -q -O - -T 3 http://web | grep title'
<title>Welcome to nginx!</title>
pod "client" deleted from default namespace
As in 2.6, kubectl may also print a few more lines, e.g. If you don't see a command prompt, try pressing enter.
or Session ended, resume using 'kubectl attach …' command. You can ignore them.
Create the file web-policy.yaml in ~/labs/5.4 with a NetworkPolicy that lets only
Pods with the label access: "true" reach the web server:
Apply it and call the web server again, once without and once with the label:
kubectl apply -f web-policy.yaml
networkpolicy.networking.k8s.io/web created
kubectl run client --image=busybox:1.37 --rm -it --restart=Never -- sh -c 'wget -q -O - -T 3 http://web | grep title'
wget: can't connect to remote host (10.43.37.199): Connection refused
pod "client" deleted from default namespace
pod default/client terminated (Error)
<title>Welcome to nginx!</title>
pod "client" deleted from default namespace
Explanation
Without a NetworkPolicy, every Pod can reach every other Pod. As soon as a policy selects a Pod (here with
podSelector all Pods with app: web), only the traffic the policy allows gets in: connections from Pods with the
label access: "true" in the same namespace. --labels gives the temporary Pod this label. In the manifest, the
quotes make true a string: label values are always strings.
In your cluster, k3s enforces the policy with its built-in network policy controller and refuses the connection right
away. wget fails, so the temporary Pod ends with Error. Other network plugins may drop the connection instead,
so the client runs into its timeout of three seconds (-T 3).
Optional Tasks
Finished early? Try these tasks. The following labs don’t depend on them. Each task lists the commands you need.
Open the solution if you get stuck.
Task 1: Inside Job
Run a Pod inside with the ServiceAccount reader and the image registry.k8s.io/kubectl:v1.35.5, whose container
runs kubectl get pods,secrets. What does its log show?
The field serviceAccountName in the Pod spec sets the ServiceAccount. The image starts kubectl with the args
of the container, e.g. args: ["get", "pods,secrets"]. You know all the commands you need.
NAME READY STATUS RESTARTS AGE
inside 0/1 ContainerCreating 0 2s
secure 1/1 Running 0 18s
web-f9b8bfb4c-gnbk6 1/1 Running 0 15s
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:reader" cannot list resource "secrets" in API group "" in the namespace "default"
kubectl inside the Pod used the token of the ServiceAccount, which Kubernetes mounts into every Pod. The API server
applied the Role from Step 1: the Pods were listed, the Secrets were forbidden. That’s why the Pod ended with the
status Error.
Task 2: Warning Shot
Create a namespace warned that only warns about Pods that violate the standard restricted, instead of rejecting
them. Start the busybox Pod from Step 2 in it.
The label key is pod-security.kubernetes.io/warn. You know all the commands you need.
kubectl run plain --image=busybox:1.37 -n warned --rm -it --restart=Never -- id
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "plain" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "plain" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "plain" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "plain" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
uid=0(root) gid=0(root) groups=0(root),10(wheel)
pod "plain" deleted from warned namespace
The Pod ran as root, but you got the same list as in Step 3 as a warning. That’s how you introduce a standard: first
warn, fix the workloads that show up, then enforce.
Task 3: Wrong Neighborhood
Create a namespace other and call the web server from a Pod with the label access=true in it. Why is the
connection refused? Allow it with a second NetworkPolicy, without changing the first one.
A Pod in another namespace reaches the Service as web.default. In the from part of a policy, a
namespaceSelector selects namespaces. Every namespace carries the label kubernetes.io/metadata.name with its name.
You know all the commands you need.
Solution
Create the file other.yaml in ~/labs/5.4 and apply it:
apiVersion:v1kind:Namespacemetadata:name:other
kubectl apply -f ~/labs/5.4/other.yaml
namespace/other created
kubectl run client -n other --labels=access=true --image=busybox:1.37 --rm -it --restart=Never -- sh -c 'wget -q -O - -T 3 http://web.default | grep title'
wget: can't connect to remote host (10.43.37.199): Connection refused
pod "client" deleted from other namespace
pod other/client terminated (Error)
The podSelector in from only matches Pods in the namespace of the policy. Create the file
web-policy-other.yaml in ~/labs/5.4:
networkpolicy.networking.k8s.io/web-from-other created
Run the client in other again:
kubectl run client -n other --labels=access=true --image=busybox:1.37 --rm -it --restart=Never -- sh -c 'wget -q -O - -T 3 http://web.default | grep title'
<title>Welcome to nginx!</title>
pod "client" deleted from other namespace
namespaceSelector and podSelector in the same entry must both match: Pods with access: "true" in the namespace
other. NetworkPolicies add up: the web server now accepts what either policy allows.
Clean Up
Delete all resources of this lab, including the namespaces:
kubectl delete -f ~/labs/5.4/
serviceaccount "reader" deleted from default namespace
role.rbac.authorization.k8s.io "pod-reader" deleted from default namespace
rolebinding.rbac.authorization.k8s.io "reader-pod-reader" deleted from default namespace
namespace "restricted" deleted
pod "secure" deleted from default namespace
networkpolicy.networking.k8s.io "web" deleted from default namespace
deployment.apps "web" deleted from default namespace
service "web" deleted from default namespace
It takes about ten seconds, because deleting a namespace waits for its Pods. If you did the optional tasks, the
output also lists inside, other, warned and web-from-other.