RKE2 fails to upgrade to 1.27.16+rke2r2,

[sdiamond2]

Environmental Info:
RKE2 Version: 1.26.11+rke2r1 (current)
I am currently looking at upgrading up to v1.29.11+rke2r1

Node(s) CPU architecture, OS, and Version:

Ubuntu 22.04, kernel 5.15.0-126-generic

Cluster Configuration:

Running 3 control plane in my test environment.

Describe the bug:

I have an airgap environment. Which means I need to use a private registry. In previous version doing the registry rewrite worked totally fine. But once I tried to upgrade to 1.27.16+rke2r2, it started to fail to get images from my private registry.

I am doing a rewrite such as this:

I also set system-default-registry = "XXXXXX"

  registry_mirrors = [
    {
      hostname = "XXXXXX"
      endpoints = [
        "XXXXXX"
      ]
      rewrites = {
        "^rancher/(.*)" = "library/rancher/$1"
      }
    }
  ]

This is failing but seems like the latest image of containerd has the fix. But is not available for 1.27 and neither 1.28 version of rke2.

I am having a similar error as k3s-io/k3s#11191

I can upgrade to 1.27.9 without any issue, but it seems to fail if I am using a newer version of Containerd.

[brandond]

I'm not sure exactly what's going on here as that is not the correct structure for registries.yaml, and you've redacted other bits of information in ways that make it hard to tell what you're intending to have configured.

I suspect that you're affected by an intentional change in behavior. The RKE2 docs haven't been updated yet, but see
https://docs.k3s.io/installation/private-registry#rewrites

Rewrites are no longer applied to the Default Endpoint as of the January 2024 releases: v1.26.13+k3s1, v1.27.10+k3s1, v1.28.6+k3s1, v1.29.1+k3s1
Prior to these releases, rewrites were also applied to the default endpoint, which would prevent K3s from pulling from the upstream registry if the image could not be pulled from a mirror endpoint, and the image was not available under the modified name in the upstream.

That page contains a workaround.

Honestly I would probably suggest not setting system-default-registry at all, and just configure your registry as a mirror for docker.io in registries.yaml. When using your registry as a mirror for another registry, the rewrites will still be applied.

[sdiamond2]

sounds good. I will try all of this and report back.

Thank you for the additional information.