From ea58d7038c275de97e677fb7a2677bc373df43c8 Mon Sep 17 00:00:00 2001 From: Derek Nola Date: Fri, 9 Aug 2024 13:02:12 -0700 Subject: [PATCH] Update hardening guide with new Manual remediations Signed-off-by: Derek Nola --- docs/security/hardening-guide.md | 305 ++++++++----------------------- 1 file changed, 77 insertions(+), 228 deletions(-) diff --git a/docs/security/hardening-guide.md b/docs/security/hardening-guide.md index b8870ae16..5208d2e63 100644 --- a/docs/security/hardening-guide.md +++ b/docs/security/hardening-guide.md @@ -40,7 +40,7 @@ By default, K3s does not include any pod security or network policies. However, ### Pod Security - + K3s v1.25 and newer support [Pod Security Admissions (PSAs)](https://kubernetes.io/docs/concepts/security/pod-security-admission/) for controlling pod security. PSAs are enabled by passing the following flag to the K3s server: @@ -533,19 +533,28 @@ rules: Both configurations must be passed as arguments to the API Server as: + + + +```yaml +kube-apiserver-arg: + - 'admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml' + - 'audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' + - 'audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml' + - 'audit-log-maxage=30' + - 'audit-log-maxbackup=10' + - 'audit-log-maxsize=100' +``` + + + ```bash --kube-apiserver-arg='audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' --kube-apiserver-arg='audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml' ``` + -If the configurations are created after K3s is installed, they must be added to K3s' systemd service in `/etc/systemd/system/k3s.service`. - -```bash -ExecStart=/usr/local/bin/k3s \ - server \ - '--kube-apiserver-arg=audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' \ - '--kube-apiserver-arg=audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml' \ -``` + K3s must be restarted to load the new configuration. @@ -560,13 +569,14 @@ sudo systemctl restart k3s.service The configuration below should be placed in the [configuration file](../installation/configuration.md#configuration-file), and contains all the necessary remediations to harden the Kubernetes components. - + ```yaml protect-kernel-defaults: true secrets-encryption: true kube-apiserver-arg: + - "enable-admission-plugins=NodeRestriction,EventRateLimit" - 'admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml' - 'audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' - 'audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml' @@ -575,10 +585,9 @@ kube-apiserver-arg: - 'audit-log-maxsize=100' kube-controller-manager-arg: - 'terminated-pod-gc-threshold=10' - - 'use-service-account-credentials=true' kubelet-arg: - 'streaming-connection-idle-timeout=5m' - - 'make-iptables-util-chains=true' + - "tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305" ``` @@ -597,254 +606,94 @@ kube-apiserver-arg: - 'audit-log-maxsize=100' kube-controller-manager-arg: - 'terminated-pod-gc-threshold=10' - - 'use-service-account-credentials=true' kubelet-arg: - 'streaming-connection-idle-timeout=5m' - 'make-iptables-util-chains=true' + - "tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305" ``` +## Manual Operations +The following are controls that K3s currently does not pass by with the above configuration applied. These controls require manual intervention to fully comply with the CIS Benchmark. -## Control Plane Execution and Arguments - -Listed below are the K3s control plane components and the arguments they are given at start, by default. Commented to their right is the CIS 1.6 control that they satisfy. - -```bash -kube-apiserver - --advertise-port=6443 - --allow-privileged=true - --anonymous-auth=false # 1.2.1 - --api-audiences=unknown - --authorization-mode=Node,RBAC - --bind-address=127.0.0.1 - --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs - --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt # 1.2.31 - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy # 1.2.17 - --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt # 1.2.32 - --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt # 1.2.29 - --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key # 1.2.29 - --etcd-servers=https://127.0.0.1:2379 - --insecure-port=0 # 1.2.19 - --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt - --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt - --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key - --profiling=false # 1.2.21 - --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt - --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key - --requestheader-allowed-names=system:auth-proxy - --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt - --requestheader-extra-headers-prefix=X-Remote-Extra- - --requestheader-group-headers=X-Remote-Group - --requestheader-username-headers=X-Remote-User - --secure-port=6444 # 1.2.20 - --service-account-issuer=k3s - --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key # 1.2.28 - --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key - --service-cluster-ip-range=10.43.0.0/16 - --storage-backend=etcd3 - --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt # 1.2.30 - --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key # 1.2.30 - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 -``` - -```bash -kube-controller-manager - --address=127.0.0.1 - --allocate-node-cidrs=true - --bind-address=127.0.0.1 # 1.3.7 - --cluster-cidr=10.42.0.0/16 - --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt - --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key - --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig - --port=10252 - --profiling=false # 1.3.2 - --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt # 1.3.5 - --secure-port=0 - --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key # 1.3.4 - --use-service-account-credentials=true # 1.3.3 -``` - -```bash -kube-scheduler - --address=127.0.0.1 - --bind-address=127.0.0.1 # 1.4.2 - --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig - --port=10251 - --profiling=false # 1.4.1 - --secure-port=0 -``` +### Control 1.1.20 +Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) +
+Remediation +K3s PKI certificate files are stored in `/var/lib/rancher/k3s/server/tls/` with permission 644. +To remediate, run the following command: ```bash -kubelet - --address=0.0.0.0 - --anonymous-auth=false # 4.2.1 - --authentication-token-webhook=true - --authorization-mode=Webhook # 4.2.2 - --cgroup-driver=cgroupfs - --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt # 4.2.3 - --cloud-provider=external - --cluster-dns=10.43.0.10 - --cluster-domain=cluster.local - --cni-bin-dir=/var/lib/rancher/k3s/data/223e6420f8db0d8828a8f5ed3c44489bb8eb47aa71485404f8af8c462a29bea3/bin - --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d - --container-runtime-endpoint=/run/k3s/containerd/containerd.sock - --container-runtime=remote - --containerd=/run/k3s/containerd/containerd.sock - --eviction-hard=imagefs.available<5%,nodefs.available<5% - --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% - --fail-swap-on=false - --healthz-bind-address=127.0.0.1 - --hostname-override=hostname01 - --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig - --kubelet-cgroups=/systemd/system.slice - --node-labels= - --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests - --protect-kernel-defaults=true # 4.2.6 - --read-only-port=0 # 4.2.4 - --resolv-conf=/run/systemd/resolve/resolv.conf - --runtime-cgroups=/systemd/system.slice - --serialize-image-pulls=false - --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt # 4.2.10 - --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key # 4.2.10 +chmod -R 600 /var/lib/rancher/k3s/server/tls/*.crt ``` -Additional information about CIS requirements 1.2.22 to 1.2.25 is presented below. - -## Known Issues -The following are controls that K3s currently does not pass by default. Each gap will be explained, along with a note clarifying whether it can be passed through manual operator intervention, or if it will be addressed in a future release of K3s. - -### Control 1.2.15 -Ensure that the admission control plugin `NamespaceLifecycle` is set. -
-Rationale -Setting admission control policy to `NamespaceLifecycle` ensures that objects cannot be created in non-existent namespaces, and that namespaces undergoing termination are not used for creating the new objects. This is recommended to enforce the integrity of the namespace termination process and also for the availability of the newer objects. - -This can be remediated by passing this argument as a value to the `enable-admission-plugins=` and pass that to `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below. -
- -### Control 1.2.16 -Ensure that the admission control plugin `PodSecurityPolicy` is set. -
-Rationale -A Pod Security Policy is a cluster-level resource that controls the actions that a pod can perform and what it has the ability to access. The `PodSecurityPolicy` objects define a set of conditions that a pod must run with in order to be accepted into the system. Pod Security Policies are comprised of settings and strategies that control the security features a pod has access to and hence this must be used to control pod access permissions. - -This can be remediated by passing this argument as a value to the `enable-admission-plugins=` and pass that to `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below. -
- -### Control 1.2.22 -Ensure that the `--audit-log-path` argument is set. -
-Rationale -Auditing the Kubernetes API Server provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system. Even though currently, Kubernetes provides only basic audit capabilities, it should be enabled. You can enable it by setting an appropriate audit log path. - -This can be remediated by passing this argument as a value to the `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below.
-### Control 1.2.23 -Ensure that the `--audit-log-maxage` argument is set to 30 or as appropriate. -
-Rationale -Retaining logs for at least 30 days ensures that you can go back in time and investigate or correlate any events. Set your audit log retention period to 30 days or as per your business requirements. - -This can be remediated by passing this argument as a value to the `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below. -
+### Control 1.2.9 +Ensure that the admission control plugin EventRateLimit is set -### Control 1.2.24 -Ensure that the `--audit-log-maxbackup` argument is set to 10 or as appropriate.
-Rationale -Kubernetes automatically rotates the log files. Retaining old log files ensures that you would have sufficient log data available for carrying out any investigation or correlation. For example, if you have set file size of 100 MB and the number of old log files to keep as 10, you would approximate have 1 GB of log data that you could potentially use for your analysis. - -This can be remediated by passing this argument as a value to the `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below. -
- -### Control 1.2.25 -Ensure that the `--audit-log-maxsize` argument is set to 100 or as appropriate. -
-Rationale -Kubernetes automatically rotates the log files. Retaining old log files ensures that you would have sufficient log data available for carrying out any investigation or correlation. If you have set file size of 100 MB and the number of old log files to keep as 10, you would approximate have 1 GB of log data that you could potentially use for your analysis. - -This can be remediated by passing this argument as a value to the `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below. -
- -### Control 1.2.26 -Ensure that the `--request-timeout` argument is set as appropriate. -
-Rationale -Setting global request timeout allows extending the API server request timeout limit to a duration appropriate to the user's connection speed. By default, it is set to 60 seconds which might be problematic on slower connections making cluster resources inaccessible once the data volume for requests exceeds what can be transmitted in 60 seconds. But, setting this timeout limit to be too large can exhaust the API server resources making it prone to Denial-of-Service attack. Hence, it is recommended to set this limit as appropriate and change the default limit of 60 seconds only if needed. - -This can be remediated by passing this argument as a value to the `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below. +Remediation +Follow the [Kubernetes documentation](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#eventratelimit) and set the desired limits in a configuration file. +For this and other psa configuration, this documentation uses /var/lib/rancher/k3s/server/psa.yaml. +Then, edit the K3s config file /etc/rancher/k3s/config.yaml and set the below parameters. +```yaml +kube-apiserver-arg: + - "enable-admission-plugins=NodeRestriction,EventRateLimit" + - "admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml" +```
-### Control 1.2.27 -Ensure that the `--service-account-lookup` argument is set to true. -
-Rationale -If `--service-account-lookup` is not enabled, the apiserver only verifies that the authentication token is valid, and does not validate that the service account token mentioned in the request is actually present in etcd. This allows using a service account token even after the corresponding service account is deleted. This is an example of time of check to time of use security issue. +### Control 1.2.11 +Ensure that the admission control plugin AlwaysPullImages is set -This can be remediated by passing this argument as a value to the `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below. -
- -### Control 1.2.33 -Ensure that the `--encryption-provider-config` argument is set as appropriate.
-Rationale -`etcd` is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted at rest to avoid any disclosures. - -Detailed steps on how to configure secrets encryption in K3s are available in [Secrets Encryption](secrets-encryption.md). +Remediation +Permissive, per CIS guidelines, +"This setting could impact offline or isolated clusters, which have images pre-loaded and +do not have access to a registry to pull in-use images. This setting is not appropriate for +clusters which use this configuration." +Edit the K3s config file /etc/rancher/k3s/config.yaml and set the below parameter. +```yaml +kube-apiserver-arg: + - "enable-admission-plugins=...,AlwaysPullImages,..." +```
-### Control 1.2.34 -Ensure that encryption providers are appropriately configured. -
-Rationale -Where `etcd` encryption is used, it is important to ensure that the appropriate set of encryption providers is used. Currently, the `aescbc`, `kms` and `secretbox` are likely to be appropriate options. - -This can be remediated by passing a valid configuration to `k3s` as outlined above. Detailed steps on how to configure secrets encryption in K3s are available in [Secrets Encryption](secrets-encryption.md). -
+### Control 1.2.21 +Ensure that the --request-timeout argument is set as appropriate -### Control 1.3.1 -Ensure that the `--terminated-pod-gc-threshold` argument is set as appropriate.
-Rationale -Garbage collection is important to ensure sufficient resource availability and avoiding degraded performance and availability. In the worst case, the system might crash or just be unusable for a long period of time. The current setting for garbage collection is 12,500 terminated pods which might be too high for your system to sustain. Based on your system resources and tests, choose an appropriate threshold value to activate garbage collection. - -This can be remediated by passing this argument as a value to the `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below. +Remediation +Permissive, per CIS guidelines, +"it is recommended to set this limit as appropriate and change the default limit of 60 seconds only if needed". +Edit the K3s config file /etc/rancher/k3s/config.yaml +and set the below parameter if needed. For example, +```yaml +kube-apiserver-arg: + - "request-timeout=300s" +```
-### Control 3.2.1 -Ensure that a minimal audit policy is created. -
-Rationale -Logging is an important detective control for all systems, to detect potential unauthorized access. +### Control 4.2.13 +Ensure that a limit is set on pod PIDs -This can be remediated by passing controls 1.2.22 - 1.2.25 and verifying their efficacy. -
- -### Control 4.2.7 -Ensure that the `--make-iptables-util-chains` argument is set to true.
-Rationale -Kubelets can automatically manage the required changes to iptables based on how you choose your networking options for the pods. It is recommended to let kubelets manage the changes to iptables. This ensures that the iptables configuration remains in sync with pods networking configuration. Manually configuring iptables with dynamic pod network configuration changes might hamper the communication between pods/containers and to the outside world. You might have iptables rules too restrictive or too open. - -This can be remediated by passing this argument as a value to the `--kube-apiserver-arg=` argument to `k3s server`. An example can be found below. +Remediation +Decide on an appropriate level for this parameter and set it, +If using a K3s config file /etc/rancher/k3s/config.yaml, edit the file to set `podPidsLimit` to +```yaml +kubelet-arg: + - "pod-max-pids=" +```
-### Control 5.1.5 -Ensure that default service accounts are not actively used -
-Rationale -Kubernetes provides a `default` service account which is used by cluster workloads where no specific service account is assigned to the pod. - -Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. +### Control 5.X -The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments. +All the 5.X Controls are related to Kubernetes policy configuration. These controls are not enforced by K3s by default. -This can be remediated by updating the `automountServiceAccountToken` field to `false` for the `default` service account in each namespace. - -For `default` service accounts in the built-in namespaces (`kube-system`, `kube-public`, `kube-node-lease`, and `default`), K3s does not automatically do this. You can manually update this field on these service accounts to pass the control. -
+Refer to [CIS 1.8 Section 5](self-assessment-1.8.md#51-rbac-and-service-accounts) for more information on how to create and apply these policies. ## Conclusion