Add cluster hardening tests

[michal-gubricky]

This PR adds tests for K8s cluster hardening
Fixes SovereignCloudStack/issues#670

[cah-patrickthiem]

I started reviewing this PR some days ago and some tests are failing every time I try unfortunately. Even when I try different kind configurations.
failing tests:

• Authentication_Methods
• K8s_Endpoints_HTTPS
• Control_Plane_Ports_Security

I am not sure why, maybe a special cluster config is needed, at least they should work for you, so can you share your cluster configuration please?

[cah-patrickthiem]

I started reviewing this PR some days ago and some tests are failing every time I try unfortunately. Even when I try different kind configurations. failing tests:

• Authentication_Methods
• K8s_Endpoints_HTTPS
• Control_Plane_Ports_Security

I am not sure why, maybe a special cluster config is needed, at least they should work for you, so can you share your cluster configuration please?

Hm, actually the kind config file is already there. So I am more confused why this is not working for me.
Can you please check that said tests are really working on your side?

[michal-gubricky]

I started reviewing this PR some days ago and some tests are failing every time I try unfortunately. Even when I try different kind configurations. failing tests:

• Authentication_Methods
• K8s_Endpoints_HTTPS
• Control_Plane_Ports_Security

I am not sure why, maybe a special cluster config is needed, at least they should work for you, so can you share your cluster configuration please?

Hm, actually the kind config file is already there. So I am more confused why this is not working for me. Can you please check that said tests are really working on your side?

Sorry for the confusion. I would say tests are working but these 3 are not passing due to the current KinD setup. If it is mandatory that all tests need to be passed against KinD then we need to adjust the config of the kind cluster.

• Authentication_Methods
  • problem here is that based on standard:

  cluster MUST implement at least two methods for authentication. One of these MUST be Service Account Tokens,
  in order to provide full functionality to Pods. A second authentication mechanisms can be chosen depending on the 
  requirements of the provider and/or customer. 
  

  As the second it can be OIDC or token webhook, so If it is possible I can try to create some fake second authentication method if it is all right.
• K8s_Endpoints_HTTPS
  • here is the problem that I list all the k8s endpoints and then checks if they are secured via HTTPS like standard say. So should we just test k8s api endpoint or?
• Control_Plane_Ports_Security
  • based on standard, It MUST NOT be possible to access Kubernetes ports from outside the internal network hosting the Kubernetes cluster. Do you have some idea how to handle it?

[cah-patrickthiem]

I started reviewing this PR some days ago and some tests are failing every time I try unfortunately. Even when I try different kind configurations. failing tests:

• Authentication_Methods
• K8s_Endpoints_HTTPS
• Control_Plane_Ports_Security

I am not sure why, maybe a special cluster config is needed, at least they should work for you, so can you share your cluster configuration please?

Hm, actually the kind config file is already there. So I am more confused why this is not working for me. Can you please check that said tests are really working on your side?

Sorry for the confusion. I would say tests are working but these 3 are not passing due to the current KinD setup. If it is mandatory that all tests need to be passed against KinD then we need to adjust the config of the kind cluster.

• Authentication_Methods

  • problem here is that based on standard:

  cluster MUST implement at least two methods for authentication. One of these MUST be Service Account Tokens,
  in order to provide full functionality to Pods. A second authentication mechanisms can be chosen depending on the 
  requirements of the provider and/or customer. 
  

  As the second it can be OIDC or token webhook, so If it is possible I can try to create some fake second authentication method if it is all right.

• K8s_Endpoints_HTTPS

  • here is the problem that I list all the k8s endpoints and then checks if they are secured via HTTPS like standard say. So should we just test k8s api endpoint or?

• Control_Plane_Ports_Security

  • based on standard, It MUST NOT be possible to access Kubernetes ports from outside the internal network hosting the Kubernetes cluster. Do you have some idea how to handle it?

Alright, I see, I will try to test again against a real K8s cluster.

[cah-patrickthiem]

I was able to let all tests pass against a custom KubeAdm cluster but I needed to make some changes to the cluster, as well as to the code (see another comment).
Just for the record, here is the list:

name: scs-kaas-conformance
status: passed
meta:
  type: summary
items:
  - name: out.json
    status: passed
    meta:
      file: results/global/out.json
      type: file
    items:
      - name: Test_scs_0200_smoke
        status: passed
      - name: Test_scs_0217_sonobuoy_Kubelet_ReadOnly_Port_Disabled/kubelet_security/Kubelet_read-only_port_(10255)_should_be_disabled
        status: passed
      - name: Test_scs_0217_sonobuoy_Kubelet_ReadOnly_Port_Disabled/kubelet_security
        status: passed
      - name: Test_scs_0217_sonobuoy_Kubelet_ReadOnly_Port_Disabled
        status: passed
      - name: Test_scs_0217_sonobuoy_Control_Plane_Ports_Security/control_plane_security/Control_plane_ports_(other_than_API_server_and_NodePorts)_must_not_be_accessible_externally
        status: passed
      - name: Test_scs_0217_sonobuoy_Control_Plane_Ports_Security/control_plane_security
        status: passed
      - name: Test_scs_0217_sonobuoy_Control_Plane_Ports_Security
        status: passed
      - name: Test_scs_0217_sonobuoy_K8s_Endpoints_HTTPS/Kubernetes_endpoint_security/All_Kubernetes_endpoints_should_be_secured_via_HTTPS
        status: passed
      - name: Test_scs_0217_sonobuoy_K8s_Endpoints_HTTPS/Kubernetes_endpoint_security
        status: passed
      - name: Test_scs_0217_sonobuoy_K8s_Endpoints_HTTPS
        status: passed
      - name: Test_scs_0217_sonobuoy_Kubelet_HTTPS_Anonymous_Auth_Disabled/kubelet_security/Kubelet_HTTPS_anonymous_access_should_be_disabled_based_on_the_configz
        status: passed
      - name: Test_scs_0217_sonobuoy_Kubelet_HTTPS_Anonymous_Auth_Disabled/kubelet_security
        status: passed
      - name: Test_scs_0217_sonobuoy_Kubelet_HTTPS_Anonymous_Auth_Disabled
        status: passed
      - name: Test_scs_0217_sonobuoy_Kubelet_Webhook_Authorization_Enabled/kubelet_security/Kubelet_authorization_mode_should_be_Webhook_from_Sonobuoy_results
        status: passed
      - name: Test_scs_0217_sonobuoy_Kubelet_Webhook_Authorization_Enabled/kubelet_security
        status: passed
      - name: Test_scs_0217_sonobuoy_Kubelet_Webhook_Authorization_Enabled
        status: passed
      - name: Test_scs_0217_sonobuoy_NodeRestriction_Admission_Controller_Enabled_in_KubeAPIServer/kube-apiserver_admission_plugins/Check_if_NodeRestriction_admission_plugin_is_enabled_in_kube-apiserver_pods
        status: passed
      - name: Test_scs_0217_sonobuoy_NodeRestriction_Admission_Controller_Enabled_in_KubeAPIServer/kube-apiserver_admission_plugins
        status: passed
      - name: Test_scs_0217_sonobuoy_NodeRestriction_Admission_Controller_Enabled_in_KubeAPIServer
        status: passed
      - name: Test_scs_0217_sonobuoy_PodSecurity_Standards_And_Admission_Controller_Enabled/pod_security_standards/Pod_security_admission_controller_should_be_enabled_and_enforce_Baseline/Restricted_policies
        status: passed
      - name: Test_scs_0217_sonobuoy_PodSecurity_Standards_And_Admission_Controller_Enabled/pod_security_standards
        status: passed
      - name: Test_scs_0217_sonobuoy_PodSecurity_Standards_And_Admission_Controller_Enabled
        status: passed
      - name: Test_scs_0217_sonobuoy_Authorization_Methods/authorization_methods/At_least_two_authorization_methods_must_be_set,_one_of_which_must_be_Node_authorization_and_another_one_consisting_of_either_ABAC,_RBAC,_or_Webhook_authorization
        status: passed
      - name: Test_scs_0217_sonobuoy_Authorization_Methods/authorization_methods
        status: passed
      - name: Test_scs_0217_sonobuoy_Authorization_Methods
        status: passed
      - name: Test_scs_0217_sonobuoy_Authentication_Methods/authentication_methods/At_least_two_authentication_methods_must_be_enabled,_one_of_which_MUST_be_Service_Account_Tokens
        status: passed
      - name: Test_scs_0217_sonobuoy_Authentication_Methods/authentication_methods
        status: passed
      - name: Test_scs_0217_sonobuoy_Authentication_Methods
        status: passed
      - name: Test_scs_0217_etcd_tls_communication/etcd_security/Communication_with_etcd_MUST_be_secured_with_TLS_for_both_peer-_and_cluster-communication
        status: passed
      - name: Test_scs_0217_etcd_tls_communication/etcd_security
        status: passed
      - name: Test_scs_0217_etcd_tls_communication
        status: passed
      - name: Test_scs_0217_etcd_isolation/etcd_security/ETCD_should_be_isolated_from_the_Kubernetes_cluster
        status: passed
      - name: Test_scs_0217_etcd_isolation/etcd_security
        status: passed
      - name: Test_scs_0217_etcd_isolation
        status: passed

[cah-patrickthiem]

Some implementation notes for things I needed to configure:

For Test "Test_scs_0217_sonobuoy_PodSecurity_Standards_And_Admission_Controller_Enabled":

Two things needed to be changed in the cluster:

Enable PodSecurity admission controller:

yamlCopy# Edit /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
  containers:
  - command:
    - kube-apiserver
    - --enable-admission-plugins=NodeRestriction,PodSecurity  # Add PodSecurity

Add enforce policy annotations to namespaces:

bashCopy# For each namespace (replace <namespace>):
kubectl annotate namespace <namespace> \
    pod-security.kubernetes.io/enforce=baseline \
    pod-security.kubernetes.io/enforce-version=latest

For system namespaces, restrict the following:

kubectl annotate namespace kube-system \
    pod-security.kubernetes.io/enforce=restricted \
    pod-security.kubernetes.io/enforce-version=latest

Enable Pod Security:

chmod +x enable-pod-security.sh

(See file content below)

sudo ./enable-pod-security.sh

#enable-pod-security.sh
#!/bin/bash

# Use correct kubeconfig
export KUBECONFIG=/etc/kubernetes/admin.conf

# Enable PodSecurity admission plugin
APISERVER_MANIFEST="/etc/kubernetes/manifests/kube-apiserver.yaml"
sed -i '/--enable-admission-plugins/s/NodeRestriction/NodeRestriction,PodSecurity/' $APISERVER_MANIFEST

# Wait for API server to be ready
echo "Waiting for API server to be ready..."
until kubectl get nodes &>/dev/null; do
    sleep 5
done
echo "API server is ready"

# Add enforce policy to namespaces
for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
    echo "Setting pod security policy for namespace: $ns"
    if [[ "$ns" == "kube-system" ]]; then
        kubectl annotate namespace $ns \
            pod-security.kubernetes.io/enforce=restricted \
            pod-security.kubernetes.io/enforce-version=latest --overwrite
    else
        kubectl annotate namespace $ns \
            pod-security.kubernetes.io/enforce=baseline \
            pod-security.kubernetes.io/enforce-version=latest --overwrite
    fi
done

For Test "Test_scs_0217_sonobuoy_Authentication_Methods", I needed to use another auth method.
Enable OIDC authentication in kube-apiserver config:

# /etc/kubernetes/manifests/kube-apiserver.yaml
- --oidc-issuer-url=https://localhost:8443
- --oidc-client-id=kubernetes
- --oidc-username-claim=sub

[cah-patrickthiem]

Overall great work! Please address my comments and we can merge it soon!

[michal-gubricky]

Overall great work! Please address my comments and we can merge it soon!

Thanks for the review, I will try to address the comments as soon as possible.

[cah-patrickthiem]

unfortunately since kind and kubeadm are different things, we cannot leave the configuration here.
Imo this needs to be addressed in the implementation node section in another issue/PR. Please exclude it here.

[michal-gubricky]

I included these settings because they are necessary to ensure that certain tests pass against kind. However, if you'd prefer to address this configuration in the implementation node section in another PR/issue, I will remove it from this PR as you suggested.

[cah-patrickthiem]

Please address the last minor thing, after that LGTM.