How can I manually upgrade a cluster without causing downtime and ensure it's done properly?

[l7u7c]

Hi everyone,
I need to upgrade to the latest version RKE2 kubernetes.
I have RKE2 version v1.26.9+rke2r1 with 3 etcd nodes, 3 masters, and more than 20 workers as the picture above
So, as this documentation: https://kubernetes.io/releases/version-skew-policy/#supported-component-upgrade-order.
I need to upgrade to the latest version of 1.26, which is v1.26.12+rke2r1, then upgrade to the latest 1.27, followed by 1.28, and finally 1.29.
Could you guide me on how to manually upgrade these nodes (3 etcd, 3 masters, more than 20 workers)?
I plan to upgrade the etcd nodes first (one at a time), then the masters (one at a time), and finally the workers.
IMO, Here's my steps for upgrading to version v1.26.12+rke2r1 without has any issues:

For etcd 1:

1: Execute command: curl -sfL https://get.rke2.io/ | INSTALL_RKE2_VERSION=v1.26.12+rke2r1 sh -
2: Run systemctl restart rke2-server
then repeat these steps for etcd 2 and 3, followed by the masters 1,2,3 at a time and workers?
After all done, repeat these steps with RKE2 latest version 1.27 -> 1.28 -> 1.29.
Hope to see your answer for upgrading.

[manuelbuil]

What about using our tool to automatically upgrade: https://docs.rke2.io/upgrade/automated_upgrade?

[l7u7c]

I haven't used this because I have been operating a cluster for company use. So, I need to make sure it doesn't have any problems

[l7u7c]

Hi @manuelbuil ,
Could you give me some advice for this ?

[manuelbuil]

I haven;t done it manually, always with the SUC tool. But what you describe seems good to me

[l7u7c]

So, what happens if the upgrade fails when using the 'automatically upgrade' option?
Will the cluster be down if fails?

[manuelbuil]

You should first back everything up in case the upgrade fails and you need to restore: https://docs.rke2.io/backup_restore. It never happened to me though

[mrmuli]

@lucd077 did you end up doing the upgrade? I'm about to do a manual one too and I don't have the auto since I'm below 1.26. I thought your steps looked good (plus the backup)

[l7u7c]

@mrmuli No, I haven't. Because in my case, it's not necessary to upgrade at this moment.
Did you try to do this?

[mrmuli]

Thanks for the reply! I will be doing one soon, currently going over a bunch of compatibility checks and then I'll get to it. I configured mine with some Ansible roles, looking to take that route for the upgrades. Happy to keep you in the loop about any discoveries along the way.

[l7u7c]

Hi @mrmuli ,
I just upgraded my cluster from 1.26 to 1.29 with 3 etcd, 3 master nodes, and various worker nodes. I didn't use a tool to automatically upgrade; instead, I manually upgraded it.

The steps are very easy:

First, upgrade the cluster to the latest 1.26 version, then sequentially upgrade to 1.27, 1.28, and finally 1.29.
Steps:

1: Download the RKE2 package for the master node first with the specified version, for example:
curl -sfL https://get.rke2.io/ | INSTALL_RKE2_VERSION=v1.26.15+rke2r1 sh -
Then, sequentially restart with the command: systemctl restart rke2-server.
2: Upgrade ETCD by following the steps above.
3: Upgrade Worker nodes using the steps above, but restart with rke2-agent."
I hope this helps you

[mrmuli]

hi @lucd077 thank you for checking back! I appreciate it.

That helps indeed. Coincidentally, I was also able to follow the same procedure going from 1.22 to 1.24. I'm about to get into 1.25 with a goal of reaching 1.28, just facing some weird RKE2 psp defaults. I am glad it worked out for you!

[brandond]

If you have split-role servers I would generally recommend doing etcd -> control-plane -> agents, but it shouldn't matter too much as long as you do all the servers first.