[Release-1.29] - Failure to read certificates and key files during k3s certificate rotate-ca

[brandond]

Backport fix for Failure to read certificates and key files during k3s certificate rotate-ca

• Failure to read certificates and key files during k3s certificate rotate-ca #10689

[VestigeJ]

##Environment Details
Reproduced using VERSION=v1.29.8+k3s1
Validated using COMMIT=0dfad66a35860c252c9547a682c37ec4dd293433

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 6.4.0-150600.23.17-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"

Cluster Configuration:

NAME             STATUS   ROLES                       AGE     VERSION
ip-1-1-3-2       Ready    control-plane,etcd,master   4h11m   v1.29.8+k3s-0dfad66a

Config.yaml:

node-external-ip: 1.1.3.2
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
cluster-init: true

Reproduction

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ VERSION=v1.29.8+k3s1
$ sudo INSTALL_K3S_VERSION=$VERSION INSTALL_K3S_EXEC=server ./install-k3s.sh
$ sudo ls -lah /var/lib/rancher/k3s/server/tls/
$ sudo mkdir -p /opt/k3s/server/tls
$ openssl version | grep -qF 'OpenSSL 3' && OPENSSL_GENRSA_FLAGS=-traditional
$ sudo openssl genrsa ${OPENSSL_GENRSA_FLAGS:-} -out /opt/k3s/server/tls/service.key 2048
$ sudo cat /var/lib/rancher/k3s/server/tls/service.key >> /opt/k3s/server/tls/service.key
$ sudo ls /opt/k3s/server/tls/
$ sudo /usr/local/bin/k3s certificate rotate-ca --path=/opt/k3s/server
$ sudo journalctl -u k3s | grep -i 'certificate error'
$ sudo /usr/local/bin/k3s certificate rotate-ca --path=/opt/k3s/server --force
$ COMMIT=0dfad66a35860c252c9547a682c37ec4dd293433
$ sudo INSTALL_K3S_COMMIT=$COMMIT INSTALL_K3S_EXEC=server ./install-k3s.sh
$ sudo /usr/local/bin/k3s certificate rotate-ca --path=/opt/k3s/server

Results:

$ sudo /usr/local/bin/k3s certificate rotate-ca --path=/opt/k3s/server

WARN[0000] failed to read /opt/k3s/server/tls/server-ca.key
WARN[0000] failed to read /opt/k3s/server/tls/client-ca.crt
WARN[0000] failed to read /opt/k3s/server/tls/request-header-ca.crt
WARN[0000] failed to read /opt/k3s/server/tls/client-ca.key
WARN[0000] failed to read /opt/k3s/server/tls/request-header-ca.key
WARN[0000] failed to read /opt/k3s/server/tls/etcd/peer-ca.key
WARN[0000] failed to read /opt/k3s/server/tls/server-ca.crt
WARN[0000] failed to read /opt/k3s/server/tls/etcd/server-ca.crt
WARN[0000] failed to read /opt/k3s/server/tls/etcd/server-ca.key
WARN[0000] failed to read /opt/k3s/server/tls/etcd/peer-ca.crt
FATA[0000] see server log for details: Internal error occurred: certificate error ID 15326

$ sudo journalctl -u k3s | grep -i 'certificate error'

Sep 12 18:11:59 k3s[16973]: time="2024-09-12T18:11:59Z" level=error msg="certificate error ID 15326: failed to validate new CA certificates and keys: ETCDServerCA: new CA is self-signed, ETCDServerCAKey: new CA cert and key cannot be loaded as X590KeyPair: open /tmp/cacerts937781426/tls/etcd/server-ca.key: no such file or directory, ETCDPeerCA: new CA is self-signed, ETCDPeerCAKey: new CA cert and key cannot be loaded as X590KeyPair: open /tmp/cacerts937781426/tls/etcd/peer-ca.key: no such file or directory, ServerCA: new CA is self-signed, ServerCAKey: new CA cert and key cannot be loaded as X590KeyPair: open /tmp/cacerts937781426/tls/server-ca.key: no such file or directory, ClientCA: new CA is self-signed, ClientCAKey: new CA cert and key cannot be loaded as X590KeyPair: open /tmp/cacerts937781426/tls/client-ca.key: no such file or directory, RequestHeaderCA: new CA is self-signed, RequestHeaderCAKey: new CA cert and key cannot be loaded as X590KeyPair: open /tmp/cacerts937781426/tls/request-header-ca.key: no such file or directory"
Sep 12 18:11:59 k3s[16973]: time="2024-09-12T18:11:59Z" level=error msg="Sending HTTP 500 response to 127.0.0.1:44526: certificate error ID 15326"

--force does seem to workaround this issue
$ sudo /usr/local/bin/k3s certificate rotate-ca --path=/opt/k3s/server --force

WARN[0000] failed to read /opt/k3s/server/tls/etcd/peer-ca.key
WARN[0000] failed to read /opt/k3s/server/tls/server-ca.key
WARN[0000] failed to read /opt/k3s/server/tls/client-ca.key
WARN[0000] failed to read /opt/k3s/server/tls/request-header-ca.key
WARN[0000] failed to read /opt/k3s/server/tls/etcd/peer-ca.crt
WARN[0000] failed to read /opt/k3s/server/tls/server-ca.crt
WARN[0000] failed to read /opt/k3s/server/tls/etcd/server-ca.crt
WARN[0000] failed to read /opt/k3s/server/tls/client-ca.crt
WARN[0000] failed to read /opt/k3s/server/tls/request-header-ca.crt
WARN[0000] failed to read /opt/k3s/server/tls/etcd/server-ca.key
certificates saved to datastore

$ COMMIT=0dfad66a35860c252c9547a682c37ec4dd293433
$ sudo INSTALL_K3S_COMMIT=$COMMIT INSTALL_K3S_EXEC=server ./install-k3s.sh

[INFO]  Using commit 0dfad66a35860c252c9547a682c37ec4dd293433 as release
[INFO]  Downloading hash https://k3s-ci-builds.s3.amazonaws.com/k3s-0dfad66a35860c252c9547a682c37ec4dd293433.sha256sum
[INFO]  Downloading binary https://k3s-ci-builds.s3.amazonaws.com/k3s-0dfad66a35860c252c9547a682c37ec4dd293433
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, already exists
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

$ sudo /usr/local/bin/k3s certificate rotate-ca --path=/opt/k3s/server

WARN[0000] failed to stat ETCDPeerCAKey: stat /opt/k3s/server/tls/etcd/peer-ca.key: no such file or directory
WARN[0000] failed to stat ClientCA: stat /opt/k3s/server/tls/client-ca.crt: no such file or directory
WARN[0000] failed to stat RequestHeaderCAKey: stat /opt/k3s/server/tls/request-header-ca.key: no such file or directory
WARN[0000] failed to stat ETCDPeerCA: stat /opt/k3s/server/tls/etcd/peer-ca.crt: no such file or directory
WARN[0000] failed to stat ServerCAKey: stat /opt/k3s/server/tls/server-ca.key: no such file or directory
WARN[0000] failed to stat ClientCAKey: stat /opt/k3s/server/tls/client-ca.key: no such file or directory
WARN[0000] failed to stat RequestHeaderCA: stat /opt/k3s/server/tls/request-header-ca.crt: no such file or directory
WARN[0000] failed to stat ETCDServerCAKey: stat /opt/k3s/server/tls/etcd/server-ca.key: no such file or directory
WARN[0000] failed to stat ETCDServerCA: stat /opt/k3s/server/tls/etcd/server-ca.crt: no such file or directory
WARN[0000] failed to stat ServerCA: stat /opt/k3s/server/tls/server-ca.crt: no such file or directory
certificates saved to datastore