kubeconfig connect failed

[godmaster35]

Hi all,

our cluster ran for exactly one year and now I can no longer log in with the previously used kubeconfig

after searching around i run the following to figure out the expiration of my local kubeconfig cert
grep 'client-certificate-data' ./kubeconfig | awk '{print $2}' | base64 -d | openssl x509 -text

ok that might explain the root cause.

Can anyone please help and tell me how to fix this issue

thanks in advance..

[vitobotta]

Hi, looks like you need to rotate the certificates, see https://docs.k3s.io/cli/certificate

I thought that they had changed the validity to 10 years but according to that page it's still only 1 year. Try the commands in that page and let me know how it goes.

[godmaster35]

Hallo vitobotta,

thanks for the quick response...

yes i was at that point already but skipped it because after updating my local kubeconfig with the one from the master1 node /etc/rancher/k3s/k3s.yaml i could at least connect again to the cluster. But the both worker nodes were not working properly anymore. Half of the pods were terminating.

So I connected per ssh to master node 1 and followed the k3s docu on cert rotation. I cannot say if this changed anything because the worker nodes were still not accessible using kubectl. So I tried to drain what also not worked without a lot of error messages and than simply just rebooted both worker nodes.
Now the cluster is running again but I am already dreading the next event in a year ;-).

Is there any configuration to let k3s use longer expiration for their certs? The docs saying the cert rotation will happen even on k3s service restart but what sense could that make if guys like me are enjoinging to run the cluster for more than a year.

[vitobotta]

What I am confused about is, did you ever upgrade k3s in that past year? If not that's the reason why you ended up with expired certificates. If you upgrade k3s every couple of months the certificate rotation is done automatically and you won't happen to be again in this kind of situation. I am converting this to a discussion since it's something related to k3s, not specific to my tool.

[vitobotta]

I have added an item on the kanban board to implement a command to more easily rotate the certificates directly with hetzner-k3s - see https://github.com/users/vitobotta/projects/4?pane=issue&itemId=69422380.

It's gonna be unlikely I will add it to v2.0 since it's not a critical feature.

[godmaster35]

Hello Vito,

I apologize for the late response. I took your recommendation regarding the k3s update to heart and performed it on both the test system and the production system to gain experience and address the certificate rotation issue.
I agree with you that this isn’t actually a problem with your tool, so closing the ticket was the right decision.
I’d like to mention that I wasn’t able to perform a k3s update using the described method on either the test or production system. While the masters were correctly updated, I had no luck with the worker nodes, even after following your instructions (Upgrading to a new version of k3s + What to do if the upgrade doesn’t go smoothly). Ultimately, the simplest approach was to rerun hetzner-k3s create cluster.config.yaml after modifying the k3s version in the yaml file

[vitobotta]

rc1 of v2 is now available and it includes many fixes and improvements. Can you help with testing? See #385 for details. Thanks

[BrutalBirdie]

Sorry do bump into this discussion, but I am currently facing a Kubernetes API: Kubernetes client certificate expires in 7 days.
I am running a hetzner-k3s cluster with 3x master nodes and 3x worker nodes with autoscale.
Also running Zabbix monitoring for the cluster.

Now master[1-3] are alerting me with Kubernetes API: Kubernetes client certificate expires in 7 days which is rather odd.
I've tried rotating the certificates many times and my search in google and other sources are depleted.
I'm scraping my head about what is going on.
Every certificate in /var/lib/rancher/k3s/server/tls and /var/lib/rancher/k3s/agent/ is not expiring before 2025.

Also, the actual client-certificate in /etc/rancher/k3s/k3s.yaml is also not expiring before 2025.
At this point, I am almost certain it has something to do with Zabbix.

When getting the api metrics myself:

kubectl proxy --port=8080 &
curl -s http://localhost:8080/metrics | grep -i --color=auto 'apiserver_client_certificate_expiration_seconds_'

apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="3600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="7200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 580
apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 580
apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 580
apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 580
apiserver_client_certificate_expiration_seconds_bucket{le="7.776e+06"} 2054
apiserver_client_certificate_expiration_seconds_bucket{le="1.5552e+07"} 2054
apiserver_client_certificate_expiration_seconds_bucket{le="3.1104e+07"} 8136
apiserver_client_certificate_expiration_seconds_bucket{le="+Inf"} 12879
apiserver_client_certificate_expiration_seconds_sum 3.0446918930072504e+11
apiserver_client_certificate_expiration_seconds_count 12879

I am not so sure what to do with these values 🤔 https://kubernetes.io/docs/reference/instrumentation/metrics/

apiserver_client_certificate_expiration_seconds
Distribution of the remaining lifetime on the certificate used to authenticate a request.
Stability Level:ALPHA
Type: Histogram

Since this also states ALPHA my brain wants to discard this whole issue, but this alert nags me and I want to understand.
Also:

Alpha metrics do not have any API guarantees. These metrics must be used at your own risk, subsequent versions of Kubernetes may remove these metrics altogether, or mutate the API in such a way that breaks existing dashboards and alerts.

As far as I can see, there is no issue? Right?

[vitobotta]

Which version of k3s does you cluster use currently? The easiest way to update the certificate AFAIK is to just upgrade k3s to a newer version as that does it automatically. hetzner-k3s provides an upgrade command for this, see the docs.