Need to document steps for manual CA certificate generation / rotation

[brandond]

We need to document the steps to manually pre-generate CA certs + keys, or update the expiration on existing certs to extend the expiration. Ideally we would also output the CA hash in a format compatible with the --token arg ("K10" + SHA256 sum of cert bytes).

We don't currently have any documentation on how to do this for either k3s or rke2, but there are some good starting points at:

• How to configure my own CA for k3s ? k3s-io/k3s#1868 (comment)
• Certificate and access management for edge computing k3s-io/k3s#2342 (comment)

One important note is that the hash we are using is a SH256 sum of the server's certificate, NOT its public key. This means that any changes to the cert, including extending its expiration, will change the hash and require passing a new --token to joining clients if they are to trust it.

gz#13930
gz#15095
gz#16112

[sraillard]

For k3s, these commands generate certificates very close to those generated during a k3s v1.19.5+k3s1 fresh installation (using elliptic curves):

openssl ecparam -name prime256v1 -genkey -noout -out client-ca.key
openssl ecparam -name prime256v1 -genkey -noout -out server-ca.key
openssl ecparam -name prime256v1 -genkey -noout -out request-header-ca.key

openssl req -x509 -set_serial 0 -new -key client-ca.key         -sha256 -days 7305 -out client-ca.crt         -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-client-ca'
openssl req -x509 -set_serial 0 -new -key server-ca.key         -sha256 -days 7305 -out server-ca.crt         -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-server-ca'
openssl req -x509 -set_serial 0 -new -key request-header-ca.key -sha256 -days 7305 -out request-header-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-request-header-ca'

I have noticed two differences:

• For the original certificates, the CN is suffixed with a timestamp, looking like that: CN = k3s-client-ca@1608074122. Is-it important to add this timestamp? Can k3s work correctly without it?
• A small difference, openssl is adding the X509v3 extensions "Authority Key Identifier", but it must not be an issue.

[howels]

Please provide information to allow us to use an external CA to sign either the RKE2 CA or the individual certs.

[davidnuzik]

@brandond looks like this isn't going to make it in the milestone; moving to next.

[Martin-Weiss]

It would also be nice if we could add the information on

1. how to replace the CA after deployment
2. how to use an intermediate CA instead of a root CA

[brandond]

Wholesale replacement of the CAs isn't likely to work, since all of the tokens and certificates within the cluster end up signed using the initial CAs. You could should be able to renew or cross-sign them, but that's about it.

Use of an intermediate CA isn't currently supported, as the certificate auto-renew code will re-sign the certs using their own key (turning them back into self-signed root CAs) when they are about to expire.

[Martin-Weiss]

Yeah - I believe we might need to re-think the big picture of using / providing and managing certificates... having so many different CAs, monitoring and maintaining them / replacing them is not very user and customer friendly and I am still not sure I understand why it was build in a way where we have to have several CAs for each K8S cluster...

[brandond]

That's how Kubernetes works? Regardless of whether it's self-managed via kubeadm, or a hosted offering such as EKS - they all have multiple self-signed CA certs that are unique to that cluster.

[Martin-Weiss]

That's how Kubernetes works? Regardless of whether it's self-managed via kubeadm, or a hosted offering such as EKS - they all have multiple self-signed CA certs that are unique to that cluster.

Yes - I just do not understand the "why" as I do not see the sense behind having so many different CAs... ;-). I would expect an hierarchy with one root CA (self signed or external) and maybe some intermediates in customer environments... I might need to dig deeper...

[0xVox]

So if I followed the steps here: k3s-io/k3s#1868 (comment)

And s/k3s/rke2 in the file path, RKE2 will roll with the provided keys in that directory?

Edit, yes, the answer is yes that does work!

[galal-hussein]

This is the summary of what we offer for this feature in rke1 compared to rke2

RKE1

• rke1 provides a subcommand for generating the csrs:

rke cert generate-csr

• the csrs will contain the right CN/Org/SANS that the certs should be provisioned with
• rke1 contain a flag to point to a custom cert directory instead of generating self signed certs:

rke up --custom-certs --certs-dir <path/to/certs/dir>

K3S/RKE2

As explained in the issue, the user will need to issue the certs/keys and place them in the right places before starting rke2/k3s, as for using a custom CA (or real CA), I dont think it will fully work unless you provide the CA cert/key which is not always the case if you are using a real CA (you can only provide a certificate), reference: https://github.com/k3s-io/k3s/blob/master/pkg/daemons/control/deps/deps.go#L553-L556

[cjellick]

@galal-hussein regarding this point:

as for using a custom CA (or real CA), I dont think it will fully work unless you provide the CA cert/key which is not always the case if you are using a real CA (you can only provide a certificate)

How does this compare to RKE1? From this bit of documentation:
https://rancher.com/docs/rke/latest/en/installation/certs/#certificates
I take it that RKE1 will similarly require CA certs and keys.

[galal-hussein]

@galal-hussein regarding this point:

as for using a custom CA (or real CA), I dont think it will fully work unless you provide the CA cert/key which is not always the case if you are using a real CA (you can only provide a certificate)

How does this compare to RKE1? From this bit of documentation:
https://rancher.com/docs/rke/latest/en/installation/certs/#certificates
I take it that RKE1 will similarly require CA certs and keys.

no it just requires the CA which you can get however the key doesnt exist because it belongs to the CA, so rke2 similarly should only check for the ca certificate, however the code right now check for boths

[maximmasiutin]

There is a discussion on this topic at https://stackoverflow.com/questions/67995687/which-of-the-elliptic-curves-are-supported-for-the-kubernetes-certificates

[mahantsucf]

I have configured k3s cluster with muli-master by having own CA certs & all looks good (followed the link for the same k3s-io/k3s#1868). Rotating of certs also works after restarting k3s on all 3 masters. But what is the process to rotate CA certs, If I rotate CA certs cluster won't start due to etcd issue. Any one tried this before, Please share your experience

[katran001]

@galal-hussein @brandond Do we plan on doing anything with this for this release or move it to backlog?

[manuelbuil]

This would be extremely helpful. There is currently a user asking about this for k3s

[cwayne18]

@caroline-suse-rancher @brandond @wjimenez5271 as we've been getitng this request a lot I think it makes sense to start looking into it for inclusion into 1.26

[brandond]

Documenting custom CA certificate generation is an easy enough task, assuming we can hold on to the requirement that the certs must be generated before the cluster starts.

Supporting rotation or renewal of the cluster CAs may be slightly more difficult, as the node join workflow currently checks the hash of the certificate, and any changes (such as extending its expiration) will invalidate that and cause the node to fail to validate the cluster cert next time it restarts, if that node is using the full K10 format token. So we'll probably need to fix that first.

[brandond]

We should also make sure we cover using intermediate CAs, as described at rancher/rke2#3650.

[zackbradys]

@brandond @cwayne18 @galal-hussein Seeing if there is an update on this issue. Currently supporting a customer that is requiring for RKE2 deployment

[dereknola]

Addressed by #101