RKE2 kubelet_extra_args --tls-cipher-suites does not work

[rdb0101]

I am a devops engineer for my team and am experiencing issues with adding two new ciphers to kubelet; specifically port 10250.
Even after adding the args as suggested under "/etc/rancher/rke2/config.yaml"
kubelet_extra_args: ["--tls-cipher-suites=EDHCE-RSA-AES2560GCM-SHA384,DHE-RSA-AES256-GCM-SHA384"]
These are the changes recommended to make in the CIS Benchmark Hardening Guide; but are ineffective.
It works for the etcd_extra_args but not the kubelet_extra_args.
I also tried creating a config.yaml under "/var/lib/kubelet" however those changes were ineffective as well. Since RKE2 rke2-server manages kubelet, the environment does not support kubelet as a separate system service.

Is there anyway I can limit the kubelet_extra_args just to one cipher-suite?

[brandond]

It's kubelet-arg not kubelet_extra_args - where did you find that version of the CLI flag?

See the docs at: https://docs.rke2.io/reference/linux_agent_config

[rdb0101]

Now kubelet does not start when i make the change provided and confirmed that the rke2-server service is restarted and running

[brandond]

Please confirm exactly how you're specifying it in the config yaml or CLI flags.

I will also note that this is a very short list of cipher suites, I'm not 100% confident it will actually work.

[rdb0101]

Ok! I am adding the following:
kubelet_extra_args: ["--tls-cipher-suites=EDHCE-RSA-AES2560GCM-SHA384,DHE-RSA-AES256-GCM-SHA384"]
However it works for etcd extra args so why can't it work for kubelet? i also changed it to the following as you recommended:
kubelet-arg: ["--tls-cipher-suites=EDHCE-RSA-AES2560GCM-SHA384,DHE-RSA-AES256-GCM-SHA384"]

[brandond]

That's not the right format. You should use either:
--kubelet-arg=tls-cipher-suites=EDHCE-RSA-AES2560GCM-SHA384,DHE-RSA-AES256-GCM-SHA384 on the CLI, or if using the config file:

kubelet-arg:
  - tls-cipher-suites=EDHCE-RSA-AES2560GCM-SHA384,DHE-RSA-AES256-GCM-SHA384

But as I mentioned earlier, I suspect that this may be a way, WAY too restrictive set of allowed cipher suites, and Kubernetes will not actually function with these limitations.

[rdb0101]

SO I reattempted the recommended changes and upon restart the rke2-server system service; kubelet did not start up. Since it is a sub-component of the rke2-server service; when I removed the changes, kubelet returned to running. So I am not sure as to where I should go from here.

[rdb0101]

Oh no ! is it really? I was referencing the CIS Benchmark Self Assessment, https://docs.rke2.io/security/cis_self_assessment123 and the solution was provided.

[brandond]

If you pass invalid arguments to the kubelet, it will fail to start. Since the rest of the components run as static pods managed by the kubelet, a misconfiguration of the kubelet will prevent everything else from running as well.

[rdb0101]

Okay that makes sense, but why don't these parameters work for RKE2? (configuration wise) Sorry I don't mean to be annoying! I am just confused as to why documentation for the RKE2 CIS Kubernetes Benchmark guide does not provide a solution that is specific to RKE2 and not a different configuration; i.e. kubelet.service as a system service. I know I keep reiterating the same thing, but the Benchmark does not provide a solution for RKE2, as far as I can see. Please note that the actual RKE2 CIS Self Assessment guide states to reference the CIS Benchmark for further details on the remediation.
"Remediation: Configuration of the parameter is dependent on your use case. Please see the CIS Kubernetes Benchmark for suggestions on configuring this for your use case."

[brandond]

I am just confused as to why documentation for the RKE2 CIS Kubernetes Benchmark guide does not provide a solution that is specific to RKE2 and not a different configuration; i.e. kubelet.service as a system service.

I'm sorry, I'm not quite sure what you're looking for. We consider cipher suite selection to be operator dependent - your organization should decide for itself what are considered "Strong Cryptographic Ciphers". If that requires modifying the kubelet args to pass the --tls-cipher-suites flag to the kubelet, you can do that via the --kubelet-arg CLI flag or config file option. We use this same convention (--COMPONENT-arg) for passing flags to all of the embedded components; this is covered elsewhere in the documentation.

[rdb0101]

Hi @brandond thank you for your help and feedback. I was just looking for RKE2 documentation providing a remediation for an RKE2 airgapped environment.

[brandond]

The remediation steps shouldn't be any different for an airgapped environment. RKE2 configuration is performed identically regardless of how RKE2 is installed or how it acquires its content.

[rdb0101]

Hi @brandond the parameters you provided worked as you stated. However I am unable to remove two ciphers: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256" from the tls-cipher-suites. If you attempt to remove both of the above listed ciphers, kubelet breaks. One of the two above listed ciphers must be included in the tls-cipher-suites in order to run kubelet without it crashing. For example, I just tested only including one of the two listed ciphers and it was able to run as expected but if I removed both and only include the ciphers I am looking for it crashes. Currently the configuration I've tested has the following listed in the "/etc/rancher/rke2/config.yaml"

kubelet-arg:
   - tls-cipher-suites=TLS_RSA_WITH_AES_128_GCM_SHA256
  

I can leave either one or both of the ciphers, and it will run as expected. If I leave only one of the two and the ciphers I am looking for it works as well, as shown below:

kubelet-arg:
   - tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256

So I am able to limit the tls-cipher_suites to one, two or three ciphers. However, it has to include the 128 ciphers in order to run, without crashing.
This explains as to why kubelet kept crashing when I only included the ciphers I was limiting to, it needed one of the two 128 ciphers for kubelet to run.
Lastly, if you don't include either of the two ciphers as shown below (using the correct parameters) kubelet will crash:

kubelet-arg:
   - tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

For my use case, I need to remove the 128 ciphers in particular without kubelet continuously crashing.
What do you recommend?

[brandond]

If I try with the ones you've provided, I get the following errors:

• --kubelet-arg=tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES256_GCM_SHA384 gives:
  FATA[0003] kubelet exited: failed to construct kubelet dependencies: Cipher suite TLS_DHE_RSA_WITH_AES256_GCM_SHA384 not supported or doesn't exist
• --kubelet-arg=tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 gives:
  E0418 19:17:16.585505 53 server.go:178] "Failed to listen and serve" err="http2: TLSConfig.CipherSuites is missing an HTTP/2-required AES_128_GCM_SHA256 cipher (need at least one of TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 or TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256)

It really looks to me like what you're trying to do isn't supported by Kubernetes. Where are you getting your cipher suites list from? You must use one of the ciphers supported by golang, which are listed at https://go.dev/src/crypto/tls/cipher_suites.go - and there are no TLS_DHE_* ciphers listed whatsoever. Are you perhaps trying to adapt some instructions meant for use with another crypto stack, such as openssl?

[brandond]

I will also link to the HTTP/2 spec which (as the error above suggests) requires some of the cipher suites you're excluding: https://httpwg.org/specs/rfc7540.html#rfc.section.9.2.2

deployments of HTTP/2 that use TLS 1.2 MUST support TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 [TLS-ECDHE] with the P-256 elliptic curve [FIPS186].

[rdb0101]

hi @brandond my apologies, the only cipher I want to limit down to is the following:

kubelet_arg:
  - tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

"DHE" is not one of the six supported, as you've mentioned. So I need to just limit to the above listed cipher.

[brandond]

Right but you can't do that. Kubernetes uses HTTP/2-enabled listeners extensively for internal components, and HTTP/2 requires TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 as per the RFC. Kubernetes (or any other HTTP/2 enabled service) does not allow what you want to do.

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 is the closest you will get on a HTTP/2 endpoint.

[rdb0101]

Okay thank you for clarifying this and now I understand why this is the case. Thank you for your patience and help as well, as I know it took me awhile to come to this conclusion.