Upgrade from 1.24.x to 1.25.x doesn't replace kube-apiserver... #4224

seimic · 2023-05-12T09:34:04Z

seimic
May 12, 2023

Environmental Info:
RKE2 Version: v1.25.9+rke2r1 (842d05e)

Node(s) CPU architecture, OS, and Version: x86_64 (Ubuntu 22.04.2 LTS)

Cluster Configuration: 3 servers, 3 agents

Describe the bug:
After upgrade from 1.24.x to 1.25.x using the system-upgrade-controller some of the deployments remain untouched as it seams.
kube-apiserver, kube-controller-manager and kube-scheduler are still 1.24.x (rancher/hardened-kubernetes:v1.24.13-rke2r1-build20230412)

Steps To Reproduce:

Install Version 1.24.13+rke2r1
curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION="v1.24.13+rke2r1" sh -
Use "profile: cis-1.6" in config and start the server.
Prepare the configuration as mentioned here: https://docs.rke2.io/known_issues#hardened-125
Install system-upgrade-controller and apply an upgrade plan to version "1.25.9+rke2r1" (https://docs.rke2.io/upgrade/automated_upgrade)

Expected behavior:
All Pods with new version

Actual behavior:
Everything seems to work correctly but kube-apiserver, kube-controller-manager and kube-scheduler are still v1.24.x

Additional context / logs:
Manual upgrade, installing RKE2 1.25.x was exactly the same.
Did I miss a step in the upgrade procedure? I don't see which.

Answered by brandond

May 12, 2023

Yep, so everything's fine. The kubelet is just not updating the mirror pods for whatever reason.

View full answer

brandond · 2023-05-12T15:11:02Z

brandond
May 12, 2023
Maintainer

Please check the actual static pods manifest on disk, and the version reported in the pod logs. There is an upstream Kubernetes issue with static pods where the mirror pods visible via kubectl can be out of sync with the actual running state.

0 replies

seimic · 2023-05-12T15:52:49Z

seimic
May 12, 2023
Author

Hi,

all server nodes have this image in the manifests (in /var/lib/rancher/rke2/agent/pod-manifests): "index.docker.io/rancher/hardened-kubernetes:v1.25.9-rke2r1-build20230412"
What to do to apply this manifests? 'touch' on manifest files? They have the timestamp of the
upgrade when I see it right. Any explosions to expect by doing this?

Inside of kube-apiserver 'kubeadm version' reports the correct version.
kubeadm version: &version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.9+rke2r1", GitCommit:"a1a87a0a2bcd605820920c6b0e618a8ab7d117d4", GitTreeState:"clean", BuildDate:"2023-04-12T19:05:26Z", GoVersion:"go1.19.8 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
Rancher (its an imported cluster) recognizes the cluster as 'v1.25.9+rke2r1' too.

Kind regards,
Michael

0 replies

brandond · 2023-05-12T21:04:26Z

brandond
May 12, 2023
Maintainer

Yep, so everything's fine. The kubelet is just not updating the mirror pods for whatever reason.

1 reply

seimic May 12, 2023
Author

I like problems which turn to features short before weekend 😂
Thank you 👍🏼🙂

VarunTyagi86 · 2023-08-01T19:22:47Z

VarunTyagi86
Aug 1, 2023

I am facing the same issue post RKE2 Upgrade from 1.24.9 to 1.24.15, Is there workaround to fix this issue ?

1 reply

brandond Aug 1, 2023
Maintainer

As discussed above, it is merely cosmetic, but will be fixed in the next release. See discussions in the issues linked from #4508

uipatherthangaraje · 2023-09-22T14:04:03Z

uipatherthangaraje
Sep 22, 2023

We are facing the same issue. kube-proxy container doesn't come up on the agent node after upgrading from v1.24.8+rke2r1 to v1.25.14+rke2r1 The kubectl output shows it running with v1.24.8+rke2r1 however crictl -r unix:///run/k3s/containerd/containerd.sock ps output has no kube-proxy.

12 replies

uipatherthangaraje Sep 23, 2023

Verbose log also doesn't have any information

cat /etc/rancher/rke2/config.yaml
token: x
selinux: false
node-ip: "10.0.1.6"
kubelet-arg:
- "max-pods=500"
- "enforce-node-allocatable=pods"
- "system-reserved=cpu=300m,memory=500Mi,ephemeral-storage=1Gi"
- "kube-reserved=cpu=300m,memory=500Mi,ephemeral-storage=1Gi"
- "eviction-hard=memory.available<1Gi,nodefs.available<10%"
- "image-gc-high-threshold=70"
- "image-gc-low-threshold=50"
- "node-status-update-frequency=10s"
- "v=6"
server: https://test.test:9345

When does the error context canceled come?

grep -A2 -B2 cancel rke2-agent.service.logs
Sep 22 20:02:53 agent0 rke2[51331]: time="2023-09-22T20:02:53Z" level=info msg="Updating load balancer rke2-agent-load-balancer server addresses -> [10.0.1.5:9345 10.0.1.4:9345]"
Sep 22 20:02:53 agent0 rke2[51331]: time="2023-09-22T20:02:53Z" level=info msg="Stopped tunnel to 10.0.1.7:9345"
Sep 22 20:02:53 agent0 rke2[51331]: time="2023-09-22T20:02:53Z" level=info msg="Proxy done" err="context canceled" url="wss://10.0.1.7:9345/v1-rke2/connect"
Sep 22 20:02:56 agent0 rke2[51331]: time="2023-09-22T20:02:56Z" level=info msg="Updating load balancer rke2-api-server-agent-load-balancer server addresses -> [10.0.1.4:6443 10.0.1.5:6443 10.0.1.7:6443 sfdev4400174-cluster.infra-sf-ea.infra.uipath-dev.com:6443]"
Sep 22 20:02:56 agent0 rke2[51331]: time="2023-09-22T20:02:56Z" level=info msg="Updating load balancer rke2-agent-load-balancer server addresses -> [10.0.1.4:9345 10.0.1.5:9345 10.0.1.7:9345]"
--
Sep 22 20:08:26 agent0 rke2[51331]: time="2023-09-22T20:08:26Z" level=info msg="Updating load balancer rke2-agent-load-balancer server addresses -> [10.0.1.7:9345 10.0.1.4:9345]"
Sep 22 20:08:26 agent0 rke2[51331]: time="2023-09-22T20:08:26Z" level=info msg="Stopped tunnel to 10.0.1.5:9345"
Sep 22 20:08:26 agent0 rke2[51331]: time="2023-09-22T20:08:26Z" level=info msg="Proxy done" err="context canceled" url="wss://10.0.1.5:9345/v1-rke2/connect"
Sep 22 20:08:53 agent0 rke2[51331]: time="2023-09-22T20:08:53Z" level=info msg="Updating load balancer rke2-api-server-agent-load-balancer server addresses -> [10.0.1.7:6443 10.0.1.4:6443 10.0.1.5:6443 sfdev4400174-cluster.infra-sf-ea.infra.uipath-dev.com:6443]"
Sep 22 20:08:53 agent0 rke2[51331]: time="2023-09-22T20:08:53Z" level=info msg="Updating load balancer rke2-agent-load-balancer server addresses -> [10.0.1.5:9345 10.0.1.7:9345 10.0.1.4:9345]"
--
Sep 22 20:16:03 agent0 rke2[51331]: time="2023-09-22T20:16:03Z" level=info msg="Updating load balancer rke2-agent-load-balancer server addresses -> [10.0.1.7:9345 10.0.1.5:9345 10.0.1.4:9345]"
Sep 22 20:16:03 agent0 rke2[51331]: time="2023-09-22T20:16:03Z" level=info msg="Stopped tunnel to 10.0.1.4:9345"
Sep 22 20:16:03 agent0 rke2[51331]: time="2023-09-22T20:16:03Z" level=info msg="Proxy done" err="context canceled" url="wss://10.0.1.4:9345/v1-rke2/connect"
Sep 22 20:16:13 agent0 rke2[51331]: time="2023-09-22T20:16:13Z" level=info msg="Updating load balancer rke2-api-server-agent-load-balancer server addresses -> [10.0.1.4:6443 10.0.1.7:6443 10.0.1.5:6443 sfdev4400174-cluster.infra-sf-ea.infra.uipath-dev.com:6443]"
Sep 22 20:16:13 agent0 rke2[51331]: time="2023-09-22T20:16:13Z" level=info msg="Updating load balancer rke2-agent-load-balancer server addresses -> [10.0.1.7:9345 10.0.1.5:9345 10.0.1.4:9345]"
--
Sep 22 20:18:03 agent0 rke2[51331]: time="2023-09-22T20:18:03Z" level=info msg="Updating load balancer rke2-agent-load-balancer server addresses -> [10.0.1.7:9345 10.0.1.5:9345 10.0.1.4:9345]"
Sep 22 20:18:03 agent0 rke2[51331]: time="2023-09-22T20:18:03Z" level=info msg="Stopped tunnel to 10.0.1.4:9345"
Sep 22 20:18:03 agent0 rke2[51331]: time="2023-09-22T20:18:03Z" level=info msg="Proxy done" err="context canceled" url="wss://10.0.1.4:9345/v1-rke2/connect"
Sep 22 20:19:23 agent0 rke2[51331]: time="2023-09-22T20:19:23Z" level=info msg="Updating load balancer rke2-api-server-agent-load-balancer server addresses -> [10.0.1.4:6443 10.0.1.7:6443 10.0.1.5:6443 sfdev4400174-cluster.infra-sf-ea.infra.uipath-dev.com:6443]"
Sep 22 20:19:23 agent0 rke2[51331]: time="2023-09-22T20:19:23Z" level=info msg="Updating load balancer rke2-agent-load-balancer server addresses -> [10.0.1.5:9345 10.0.1.7:9345 10.0.1.4:9345]"
Sep 22 20:19:23 agent0 rke2[51331]: time="2023-09-22T20:19:23Z" level=info msg="Connecting to proxy" url="wss://10.0.1.4:9345/v1-rke2/connect"
Sep 22 20:23:30 agent0 systemd[1]: Stopping Rancher Kubernetes Engine v2 (agent)...
Sep 22 20:23:30 agent0 rke2[51331]: time="2023-09-22T20:23:30Z" level=info msg="Proxy done" err="context canceled" url="wss://10.0.1.5:9345/v1-rke2/connect"
Sep 22 20:23:30 agent0 rke2[51331]: time="2023-09-22T20:23:30Z" level=fatal msg="context canceled"
Sep 22 20:23:30 agent0 systemd[1]: rke2-agent.service: Main process exited, code=exited, status=1/FAILURE
Sep 22 20:23:30 agent0 systemd[1]: rke2-agent.service: Failed with result 'exit-code'.
--
Sep 22 20:23:42 agent0 rke2[778151]: time="2023-09-22T20:23:42Z" level=info msg="Tunnel authorizer set Kubelet Port 10250"
Sep 22 20:25:14 agent0 systemd[1]: Stopping Rancher Kubernetes Engine v2 (agent)...
Sep 22 20:25:14 agent0 rke2[778151]: time="2023-09-22T20:25:14Z" level=fatal msg="context canceled"
Sep 22 20:25:14 agent0 systemd[1]: rke2-agent.service: Main process exited, code=exited, status=1/FAILURE
Sep 22 20:25:14 agent0 systemd[1]: rke2-agent.service: Failed with result 'exit-code'.

I see the error Pod cannot start yet in kubelet.log

I0923 05:44:08.347213  803652 common.go:73] "Generated pod name" pod="kube-system/kube-proxy-agent0" podUID=022d8497ccc34d16ccd3332d1f1cadd0 source="/var/lib/rancher/rke2/agent/pod-manifests/kube-proxy.yaml"
I0923 05:44:08.347219  803652 common.go:78] "Set namespace for pod" pod="kube-system/kube-proxy-agent0" source="/var/lib/rancher/rke2/agent/pod-manifests/kube-proxy.yaml"
I0923 05:44:09.407873  803652 kubelet.go:2153] "SyncLoop (SYNC) pods" total=8 pods=[rook-ceph/rook-ceph-configure-script-job-cxksh rook-ceph/rook-ceph-configure-script-job-s2jgc kube-system/kube-proxy-agent0 rook-ceph/rook-ceph-configure-script-job-xrssb uipath/identity-service-clients-register-mgwdc rook-ceph/rook-ceph-configure-script-job-24xgp uipath/notificationserviceapi-5b589c4cd8-l9z4c uipath/ah-tenant-service-tenant-db-migrations-job-6bm82]
I0923 05:44:09.408024  803652 pod_workers.go:896] "Pod cannot start yet" pod="kube-system/kube-proxy-agent0" podUID=022d8497ccc34d16ccd3332d1f1cadd0
I0923 05:44:13.346060  803652 file.go:201] "Reading config file" path="/var/lib/rancher/rke2/agent/pod-manifests/kube-proxy.yaml"
I0923 05:44:13.346514  803652 common.go:73] "Generated pod name" pod="kube-system/kube-proxy-agent0" podUID=022d8497ccc34d16ccd3332d1f1cadd0 source="/var/lib/rancher/rke2/agent/pod-manifests/kube-proxy.yaml"
I0923 05:44:13.346520  803652 common.go:78] "Set namespace for pod" pod="kube-system/kube-proxy-agent0" source="/var/lib/rancher/rke2/agent/pod-manifests/kube-proxy.yaml"

brandond Sep 26, 2023
Maintainer

When does the error context canceled come?

This is a normal message to see whenever the service is stopped:

Sep 22 20:25:14 agent0 systemd[1]: Stopping Rancher Kubernetes Engine v2 (agent)...
Sep 22 20:25:14 agent0 rke2[778151]: time="2023-09-22T20:25:14Z" level=fatal msg="context canceled"

Are you stopping the service, or is something else doing that?

Can you attach the complete kubelet log? Assuming that the entire problem here isn't just that something (like rancher-system-agent perhaps) is unnecessarily restarting RKE2 before all the components can come up, there should be additional messages in the kubelet log that say why it can't start.

uipatherthangaraje Sep 26, 2023

kubelet.log
rke2-agent.service.log

brandond Sep 26, 2023
Maintainer

It looks like the kubelet log may have been rotated? the service log doesn't contain anything newer than Sep 22 20:25:35, while the kubelet log starts at 2023/09/23 05:44:01 and only contains 17 seconds of data. You may need to grab older logs and/or set the log-file-max-size kubelet arg to something larger (or 0 to disable it).

uipatherthangaraje Sep 28, 2023

Please find the new logs
containerd.log
kubelet.log
rke2-agent.service.log

We added a workaround to kill Kubelet to reconcile however it worked but sometimes it fails.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade from 1.24.x to 1.25.x doesn't replace kube-apiserver... #4224

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 14 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Upgrade from 1.24.x to 1.25.x doesn't replace kube-apiserver... #4224

seimic May 12, 2023

Replies: 5 comments · 14 replies

brandond May 12, 2023 Maintainer

seimic May 12, 2023 Author

brandond May 12, 2023 Maintainer

seimic May 12, 2023 Author

VarunTyagi86 Aug 1, 2023

brandond Aug 1, 2023 Maintainer

uipatherthangaraje Sep 22, 2023

uipatherthangaraje Sep 23, 2023

brandond Sep 26, 2023 Maintainer

uipatherthangaraje Sep 26, 2023

brandond Sep 26, 2023 Maintainer

uipatherthangaraje Sep 28, 2023

seimic
May 12, 2023

Replies: 5 comments 14 replies

brandond
May 12, 2023
Maintainer

seimic
May 12, 2023
Author

brandond
May 12, 2023
Maintainer

seimic May 12, 2023
Author

VarunTyagi86
Aug 1, 2023

brandond Aug 1, 2023
Maintainer

uipatherthangaraje
Sep 22, 2023

brandond Sep 26, 2023
Maintainer

brandond Sep 26, 2023
Maintainer