questions on k3s hardening

[cwrx777]

1. where does k3s set the default control plane execution and arguments for apiserver, controller manager, scheduler, kubelet?
2. Any reasons why the required Admission control plugins are not by default? is it supposed to be set in my config.yaml, like this ?

• NamespaceLifecycle,
• PodSecurityPolicy
• EventRateLimit
• AlwaysPullImages

3. In case I need to enable additional admission plugins or other k8s component parameters using my config.yaml, do I need to also specify the default value, e.g. NodeRestriction?
4. What does this command do?

   journalctl -D /var/log/journal  -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep

5. Is configuration for kubernetes components content outdated?
6. is RotateKubeletServerCertificate supported in v1.29?

[brandond]

Please don't open multiple discussions on the same topic. Lets keep this at k3s-io/docs#300 (comment)

[cwrx777]

@brandond
possible to use this thread instead ?

[dereknola]

1. Most are set in https://github.com/k3s-io/k3s/blob/master/pkg/daemons/control/server.go
2. K3s is not hardened by default. You should look into RKE2, which has hardening with profile:cis if you are interested in that. Also, PSP are only used on 1.24 and older. That Known Issues section needs to be cleaned up.
3. I'm not sure what you mean by this. Any additional parameters you set are on you to support.
4. K3s runs all the kubernetes control plane components in a single process. This is looking thru the process logs to find the arguments that the kube-apiserver launched with. Try and run it yourself on your nodes and you will see what I'm talking about.
5. No, you should be able to pass up through cis-1.8 with that configuration. cis-1.9 just hit kube-bench and we are working on getting K3s/RKE2 conformant with it (although not much will likely change, we just haven't 100% finished that work).
6. Yes, this check is outdated, as the rotate feature gate has been "on by default" since Kubernetes 1.11. This check is basically just "did you do something stupid and turn off the security feature".

[cwrx777]

Hi @dereknola Thank you for answering my questions. I have additoinal questions.

I'm not sure what you mean by this. Any additional parameters you set are on you to support.

After I run k3s server with config.yaml according to this, I can see that there are parameters that are set even it is not in the configuration file. and you have answered that most are set in here.
Using enable-admission-plugins as example, it is hardcoded to NodeRestriction. If I want to enable additional plugins, I believe i should put that in the k3s config.yaml.
Do I put it as
enable-admission-plugins=NodeRestriction,EventRateLimit or
enable-admission-plugins=EventRateLimit ?

Yes, this check is outdated, as the rotate feature gate has been "on by default" since Kubernetes 1.11. This check is basically just "did you do something stupid and turn off the security feature".

how do I audit that RotateKubeletServerCertificate feature is enabled?

Additional questions:

• 1.1.13
  why is this Not Applicable ? isn't it just changing the file permission to 600?

My worker node is run using the following config.yaml:

...
protect-kernel-defaults: true
kubelet-arg:
  - 'streaming-connection-idle-timeout=5m'
  - 'make-iptables-util-chains=true'

k3s-agent.service:

ExecStart=/usr/local/bin/k3s \
    agent \
        '--config=/test/k3s/config/config.yaml' \

• 4.1.9, 4.1.10
  Does kubelet --config configuration file refer to kubelet.kubeconfig?
• 4.2.8, 4.2.10, 4.2.11
  I am not able to verify the required parameters from the ps output.

  /bin/ps -fC containerd
  UID          PID    PPID  C STIME TTY          TIME CMD
  root     3002164 3001447  0 Jul08 ?        00:06:39 containerd

• 4.2.12
  Does the kubelet config refer to /var/lib/rancher/k3s/agent/kubelet.kubeconfig
kubelet.kubeconfig:

  apiVersion: v1
  clusters:
  - cluster:
      server: https://127.0.0.1:6444
      certificate-authority: /var/lib/rancher/k3s/agent/server-ca.crt
    name: local
  contexts:
  - context:
      cluster: local
      namespace: default
      user: user
    name: Default
  current-context: Default
  kind: Config
  preferences: {}
  users:
  - name: user
    user:
      client-certificate: /var/lib/rancher/k3s/agent/client-kubelet.crt
      client-key: /var/lib/rancher/k3s/agent/client-kubelet.key
  

• 4.2.1.3
  Where is pod-pids supposed to be set? I dont see that this is set in the kubelet CLI arguments.

   Running kubelet 
  --address=0.0.0.0 
  --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding 
  --anonymous-auth=false 
  --authentication-token-webhook=true 
  --authorization-mode=Webhook 
  --cgroup-driver=systemd 
  --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt 
  --cloud-provider=external 
  --cluster-dns=10.43.0.10 
  --cluster-domain=cluster.local 
  --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock 
  --containerd=/run/k3s/containerd/containerd.sock 
  --eviction-hard=imagefs.available<5%,nodefs.available<5% 
  --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% 
  --fail-swap-on=false 
  --feature-gates=CloudDualStackNodeIPs=true 
  --healthz-bind-address=127.0.0.1 
  --hostname-override=skhromivpapp03.shses.shs.com.sg 
  --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig 
  --make-iptables-util-chains=true 
  --node-ip=10.236.49.161 
  --node-labels=node-type=worker 
  --pod-infra-container-image=rancher/mirrored-pause:3.6 
  --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests 
  --protect-kernel-defaults=true 
  --read-only-port=0 
  --resolv-conf=/etc/resolv.conf 
  --serialize-image-pulls=false 
  --streaming-connection-idle-timeout=5m 
  --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt 
  --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key