cks-short-tips

1. Have a plan for preparation
   • Curated CKS Resources
2. Train for speed
   • Disable unsafe pasting
   • Use aliases and tab completion
   • Be fast with CLI tools
3. Know your tools and technologies
4. Be thorough
5. Don't give up

Have a plan for preparation

There a tons of resources out there. Use them to acquire a wider knowledge, before training specifically for the exam.

Here's my plan, as an example:

I invested about 5 working days over a period of three weeks before the exam.
As a seasoned developer in the public cloud I was quick with kubectl, security-context and so on, but not so much with api-server config.
Retrospectively, spending maybe one more day would not have been advisable, as I had to look up a lot of api-server-related stuff in the docs during the exam.

1. Read the Docs for the Certs.
   IMO most importantly: Resources Allowed: All LF Certification Programs - T&C DOCS (Candidate Facing Resources).
2. Work your way through the curriculum.
   • read through Walid Shaari's curated resources, which add a lot of useful detail to rather abstract curriculum.
   • follow-up reading on every topic!
   • get your handy dirty! Install every tool on your local kubernetes cluster and apply the getting started docs.
     I use k3d via gitops playground
3. Solve example questions, learn from the results and from your errors.
   • Mohamed Abukar's mock exam questions
   • These look like old killer.sh Questions with solutions, but no harm in having a look at them before having you first killer.sh try.
   • These look promising, including a learning platform: ViktorUJ/cks
     I discovered them only after I passed my CKS, though.
4. Have your first try at killer.sh (the week before the exam)
   • Take it serious, but be prepared to fail.
   • Main objective: Get accustomed to the exam environment and find out what to improve
   • Note down all your weak spots, both technologically as well es regarding efficiency on the shell/keyboard
5. Have your second try at killer.sh (one or two days before the exam)
   • This should give you confidence that you're now fully prepared and very time efficient with the tools
   • Note down some more fine-tuning
   • Follow up on the things that were not perfect
6. Pass the exam
   • Begin with the first question and try to solve as many questions as possible
   • Only if you're very unsure, flag some questions and come back later.
     Note that this takes some time, because you'll likely read the question and think about it twice.
     On the other hand, losing a lot of time with one question might lead to not having enough time to complete easier tasks later.
   • Don't give up until the exam ends 💪
   • 🥂

Curated CKS Resources

• Interactive learning environments
  • Two tries for interactive killer.sh included in your exam voucher. Use them, but use them wisely!
  • If you need more, create an account as killercoda
  • ViktorUJ/cks - local platform for learning kubernetes and preparation for CKS
• Community resources
  • Walid Shaari's curated resources
  • Video Tutorials by Venkata Ramana Gali, corresponding Repo
• Example questions
  • Mohamed Abukar's mock exam questions
  • These look promising, including a learning platform: ViktorUJ/cks
  • These look like old killer.sh Questions with solutions, but no harm in having a look at them before having you first killer.sh try.
• Udemy Courses
  even when you don't take a course, having a look at the content might help discovering topics that are relevant
• There are a lot more, when you follow the links in the repos above.

Train for speed

Disable unsafe pasting

Disable "unsafe pasting" in VM terminal emulator "Terminal Preferences->General->Show unsafe paste dialog".

Use aliases and tab completion

Those are the ones I used

vi ~/.bashrc
alias kg='k get'
alias kd='k describe'
alias ka='k apply -f'
alias kn='k config set-context --current --namespace'
# : wq
source ~/.bashrc

Note:

• Tab completion does not work for aliases 😐️
  So I mainly use them to get a fast overview of pods, services, etc.

kg po
kg svc
kg ing
kg deploy

• There already is an alias k=kubectl.
  Tab completion does work for it, so use e.g. k get po <TAB> instead of typing or copying pod names
• killer.sh recommends defining these variables, but I use ctrl + z + bg instead

export do="--dry-run=client -o yaml"    # k create deploy nginx --image=nginx $do

export now="--force --grace-period 0"   # k delete pod x $now

• vi is already set up with useful options, usually no need to edit ~/.vimrc

Be fast with CLI tools

• Bash:
  • ctrl + z: keep open but return to shell
  • then fg return e.g. in vi
  • or bg keeps running in background e.g. for killing pods
• tmux - first things in an exam: edit bashrc (see above), then start tmux
  • split pane, preferable horizontal (ctrl+b+") (easier for copying with mouse)
  • move from pane to pane using ctrl+b + cursor keys
  • select mode (scrolling) ctrl+b + ] - then use search or cursor keys or page up/down for navigating Careful: ctrl+w might close tab. Better use mouse for copying.
  • search: in select mode ctrl+b+s or r for reverse n for next hit; Shift+n for previous hit
  • increase size of pane by keeping ctrl+b pressed and using cursor keys
  • ctrl+z to toggle full screen for a pane
• vim
  • w write, don't quit. Hints:
    • use other tmux pane to run ka, then come back to fix potential syntactic errors
    • or use ctrl + z to send vim to background, run ka then fg to return to vim
  • q quit, can be combined to wq
  • y = yank = copy
  • p = paste
  • d = delete = cut. If you want to delete more lines just to, e.g. dddd
  • u = undo
  • redo: ctrl + r
  • v = visual (select), use cursor keys, then y or p
  • indent: select then > or < (repeat with .)
• less
  • switch search to case-insensitive using :i
  • search using /
• ssh / scp, to access or copy files to/from nodes
• cut --delimiter ' ' --fields 9 # delimiter space, print field 9

Know your tools and technologies

• curl
  • Validate certificate: curl -vk 2>&1 https://xyz | grep -i subject
  • Accessing the Kubernetes API from a Pod Note the path to the secrets

k run debug-node --rm -it --image alpine --
'curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/default/secrets'

• kube-apiserver, kubelet, etc.
  • interne YAMLs konfigurierbar:
  • /etc/kubernetes/manifests
  • /etc/kubernetes/manifests/kube-apiserver.yaml
  • /etc/kubernetes/manifests/kube-controller-manager.yaml
  • /var/lib/kubelet/config.yaml
  • apiserver is automatically restarted when kube-apiserver.yaml is changed.
    If need be this can be forced using crictl rmp , or even using kubectl (k delete pod) on a controlplane node or using restarting the service

#SystemD
systemctl status kubelet
systemctl restart kubelet
# or system V
service kubelet restart
service kubelet status

• AppArmor
  • AppArmor profiles are specified per-container. To specify the AppArmor profile to run a Pod container with, add an annotation to the Pod's metadata
  • container.apparmor.security.beta.kubernetes.io/<container_name>: <profile_ref>
  • container.apparmor.security.beta.kubernetes.io/c1: localhost/very-secure
  • or runtime/default
  • Profile must be present on node
  • Accessing App Armor Docs ist permitted
  • See k8s docs for an example

apparmor_parser -q <<EOF
EOF
# or
scp profile.txt node:/tmp/profile.txt
#then
apparmor_parser -q profile.txt
# optional check
apparmor_status
# or just load it again
apparmor_parser -q profile.txt # Output profile alrady exists

• Seccomp
  • Kubernetes lets you automatically apply seccomp profiles loaded onto a node to your Pods and containers.
  • Enable the use of RuntimeDefault as the default seccomp profile for all workloads run the kubelet with the --seccomp-default command line flag in /var/lib/kubelet/config.yaml as in this example then systemctl restart kubelet
  • Set default profile or an explicit one as shown in docs

  securityContext:
    seccompProfile:
      type: RuntimeDefault
# or
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/audit.json

• PSA
  • PSAs are set namespace-wide
  • Try out e.g. this example: Apply predefined Pod-level security policies using PodSecurity  |  Google Kubernetes Engine (GKE)  |  Google Cloud
  • kubectl label --overwrite ns restricted-ns pod-security.kubernetes.io/enforce=restricted
  • kubectl label --overwrite ns baseline-ns pod-security.kubernetes.io/warn=baseline
• Falco
  • Might log to syslog -> cat /var/log/syslog | grep -i xyz or journalctl -fu falco | grep "xyz error"
  • Output UID and Container ID: %user.uid,%container.id see supported fields doc (allowed in exam!)

rules_file:
  - /etc/falco/falco_rules.yaml
  - /etc/falco/falco_rules.local.yaml
  - /etc/falco/rules.d # custom rules

• Tracee similar to falco, ebpf kernel events, signatures tigger alert.
• Trivy
  • trivy image alpine --severity=HIGH,CRITICAL
• Kubesec
  • kubesec scan k8s-deployment.yaml | jq
  • kubesec scan jenkins/tmp-docker-gid-grepper.yaml | jq '.[].scoring.advise'

apiVersion: v1
kind: Pod
spec:
  enableServiceLinks: false
  automountServiceAccountToken: false
  containers:
  - name: restricted
    securityContext:
      runAsNonRoot: true
      runAsUser: 100000
      runAsGroup: 100000
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      seccompProfile:
        type: RuntimeDefault
      capabilities:
        drop:
        - ALL

• Kube-bench
  • Has to run on node! Either operator, pod or binary on node
  • kube-bench run --targets=master > bench.txt
  • kube-bench run --targets=node > bench.txt
  • Using --target makes solving specific tasks much easier because there usually are a lot of results
  • Writing to file allows for comparing the results after mitigation.
  • kube-bench run --targets=master | less also works
• crictl, similar to docker but for CRI

crictl pods
crictl ps

crictl ps --pod $podid # all containers of pod, also works with grep $podid
crictl logs $containerid # see output of crcitcl ps --pod for $containerid
crictl rmp $podid #remove pod

• etcdctl: Access secrets directly: (command can be found in docs for encryption at rest):

  ETCDCTL_API=3 etcdctl \
   --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
   --cert=/etc/kubernetes/pki/etcd/server.crt \
   --key=/etc/kubernetes/pki/etcd/server.key  \
   get /registry/secrets/default/secret1 | hexdump -C

• strace Count time, calls, and errors for each system call and report a summary on program exit: strace -p pid -c or ongoing output, e.g. in separate tmux pane/window strace -p pid 2>&1 | grep -i kill e.g. syscalls of container

crictl pods # -> POD_ID
critctl ps --pod POD_ID # -> CONTAINER_ID
crictl inspect CONTAINER_ID | grep -i pid # -> PID
strace -p PID -c
# Ctrl + C 
#% time     seconds  usecs/call     calls    errors syscall
#------ ----------- ----------- --------- --------- ---------------
# 81,08    0,000030          30         1         1 rt_sigtimedwait
# 18,92    0,000007           3         2           wait4
#------ ----------- ----------- --------- --------- ----------------
#100,00    0,000037          12         3         1 total

• OPA conftest
  • conftest test some.yaml
  • looks for policies in folder policy
• OPA Gatekeeper
  • OPA = general purpose policy engine; Gatekeeper = kubernetes-specific impl of OPA as K8s Admission controller
  • ConstraintTemplate -> describe rego
  • Constraint -> describes matching rules, parameters, etc., lists violations
    • Violations of constraints are listed in the status field of the corresponding constraint
    • The audit pod emits JSON-formatted audit logs to stdout.
    • Prometheus Metrics (not relevant for CKS)
  • example:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-gk
spec:
  enforcementAction: dryrun # deny|warn

# from killer.sh
## Question 7
k get crd | grep gatekeeper
k edit blacklistimages pod-trusted-images # constraint: add parameter
k edit constrainttemplates blacklistimages #-> template: change rego expression

## Preview Question 2
# constraint, add parameter for namespace
k edit requiredlabels namespace-mandatory-labels
# template, fix rego
k edit constrainttemplates requiredlabels

• dmesg - prints kernel logs
• kubeadm

# control plane
k drain $NODE
# ssh to master
apt update && apt install kubeadm=1.29.0-1.1 # if necessary
kubeadm upgrade plan # outputs commands for apply
kubeadm upgrade apply v1.24.1 #use specific version! Outputs message to upgrade kubelet
apt install kubelet=1.29.0-1.1 kubectl=1.29.0-1.1
service kubelet restart #systemctl restart kubelet
k uncordon $NODE

# node: drain, ssh
apt update && apt install kubeadm=1.29.0-1.1
kubeadm upgrade node
service kubelet restart
# exit, k uncordon

• sha512sum

  sha512sum -c FILE # can contain all sums and binaries
  # FILE
  #SUM1  /a/file
  #SUM2  another-file
  # or indivudal files
  sha512sum -c <<< "$1  $2"

• You might want to install helpful software. Not sure if it is allowed, though
  • apt install bat-> binary is called batcat - syntax highlighted YAML files
  • or tldr if you need examples for using binaries or curl cheat.sh/cut

Be thorough

• Switch to the correct kubecontext first (there is a command to copy at the beginning of each question)
• Read questions carefully! There a several subtasks.
• Read again when done to see if you did not miss anything.
• Validate your work, e.g. k get node, k get pod, k exec -it ...
• Double check if you wrote the result to the right file
  • Sometimes it has to be written to a file on the main host
  • Sometimes it has to be written to a file on a node
  • Sometimes it is not necessary to write a result
• Every exercise features links to the relevant docs in the header. Have a look a them before searching in the docs first!
• Copy each YAML before editing to your home folder, to be able to reset after possibly messing it up.
  Don't copy it to the folder of the original file, this might confuse a running api server, for example.

In both my killer.sh tries I missed some points.
Even though I solved things correctly or would have been able to, I wrote the result into a wrong file or missed to solve a subtask.
Don't do that, be thorough!

Don't give up

• You are going to need every point to can score and every second you're granted
• If some tasks might seem to difficult, flag them and tackle them again at the end
• Having only one or two minutes left is enough time to score some points by partially solving some question
• You'll be surprised how fast you can learn to solve tasks under pressure
• If you don't know how to solve, start by reading the docs linked at the top
• You don't need to solve the whole question, partial solutions score points as well
• You never know, a partial solution saved in the last seconds might be the one point you need for the 70% passing score!

Don't give up until you're kicked out of the exam 💪