Wrong image versions used for some of the kubeadm images in 1.26

[afbjorklund]

The images for "etcd" and "pause" were not bumped correctly (for the v1.26.0).

This causes double downloads, and issues in air-gapped/firewalled installations:

docker@minikube:~$ sudo crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
gcr.io/k8s-minikube/storage-provisioner   v5                  6e38f40d628db       9.06MB
registry.k8s.io/coredns/coredns           v1.9.3              5185b96f0becf       17MB
registry.k8s.io/etcd                      3.5.5-0             4694d02f8e611       102MB
registry.k8s.io/etcd                      3.5.6-0             fce326961ae2d       103MB
registry.k8s.io/kube-apiserver            v1.26.0             a31e1d84401e6       35.3MB
registry.k8s.io/kube-controller-manager   v1.26.0             5d7c5dfd3ba18       32.2MB
registry.k8s.io/kube-proxy                v1.26.0             556768f31eb1d       21.5MB
registry.k8s.io/kube-scheduler            v1.26.0             dafd8ad70b156       17.5MB
registry.k8s.io/pause                     3.8                 4873874c08efc       336kB
registry.k8s.io/pause                     3.9                 e6f1816883972       322kB

minikube start --kubernetes-version=v1.26.0 --container-runtime=containerd

---

$ ~/.minikube/cache/linux/amd64/v1.26.0/kubeadm config images list
registry.k8s.io/kube-apiserver:v1.26.0
registry.k8s.io/kube-controller-manager:v1.26.0
registry.k8s.io/kube-scheduler:v1.26.0
registry.k8s.io/kube-proxy:v1.26.0
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.6-0
registry.k8s.io/coredns/coredns:v1.9.3
$ grep -A4 v1.26 pkg/minikube/constants/constants_kubeadm_images.go
		"v1.26": {
			"coredns/coredns": "v1.9.3",
			"etcd":            "3.5.5-0",
			"pause":           "3.8",
		},

The root cause is caching the values, and them changing over alpha/beta/rc/rel.

[afbjorklund]

This is fixed on master, just not working with release (v1.28.0)

		"v1.26.0-rc.1": {
			"coredns/coredns": "v1.9.3",
			"etcd":            "3.5.6-0",
			"pause":           "3.9",
		},
		"v1.26.0-rc.0": {
			"coredns/coredns": "v1.9.3",
			"etcd":            "3.5.5-0",
			"pause":           "3.9",
		},
		"v1.26.0-beta.0": {
			"coredns/coredns": "v1.9.3",
			"etcd":            "3.5.5-0",
			"pause":           "3.8",
		},
--
		"v1.26.0-alpha.3": {
			"coredns/coredns": "v1.9.3",
			"etcd":            "3.5.5-0",
			"pause":           "3.8",
		},
--
		"v1.26.0-alpha.2": {
			"coredns/coredns": "v1.9.3",
			"etcd":            "3.5.5-0",
			"pause":           "3.8",
		},
--
		"v1.26.0-alpha.1": {
			"coredns/coredns": "v1.9.3",
			"etcd":            "3.5.5-0",
			"pause":           "3.8",
		},

cc @spowelljr

[afbjorklund]

Unfortunately it is also running the wrong "pause" version, due to setting it up in the containerd config.

docker@minikube:~$ sudo ctr -n k8s.io c ls
CONTAINER                                                           IMAGE                                              RUNTIME                  
196468e32ab9e0e8b197db7871691fa5f5cda34fdb7e81adfbceac10b81aff64    registry.k8s.io/coredns/coredns:v1.9.3             io.containerd.runc.v2    
2219c2daaee7b362499fe9e378e31bf6386a7f4caea649f942f6fd102e175184    docker.io/kindest/kindnetd:v20221004-44d545d1      io.containerd.runc.v2    
2ea09c13d63c5e6bf141d203d92a7cbdaeba85f4cc7d761a61caa11f0bc8c269    registry.k8s.io/pause:3.8                          io.containerd.runc.v2    
3566b7e1e1748a279690beb37948b750e4c01e357f6c04cfc08e5967c74cf446    registry.k8s.io/pause:3.8                          io.containerd.runc.v2    
35eef4c1f74f65311e10f42f803ad4c0fe7fd19569227ce8f63d81d685d832b0    registry.k8s.io/kube-apiserver:v1.26.0             io.containerd.runc.v2    
551b44457ead8838d7c0ac87b0209c9741cecd9ee7e7e3bd71a62aebe35ade55    registry.k8s.io/pause:3.8                          io.containerd.runc.v2    
5c17df7d0f4f4221304ee1f68aae8c87a870bfb20c798c77ce79dd36ee260f05    gcr.io/k8s-minikube/storage-provisioner:v5         io.containerd.runc.v2    
79c1198d78f3f8e16004d73ec497ac3c5ea2d98ccc513e68a5b9c6f98669b71e    registry.k8s.io/pause:3.8                          io.containerd.runc.v2    
7cfc200da51c9274435e43d51c5d2fccde3ba65b2822e332e6bca529bbe15308    registry.k8s.io/kube-controller-manager:v1.26.0    io.containerd.runc.v2    
908265b0c4203187686c7b2e2ac153c99d50f983ba82b53bb6329acb14fb93f3    registry.k8s.io/pause:3.8                          io.containerd.runc.v2    
a0e16f5716a04ce38651b2a6d88ef69272bfbecc39168cfbe620f3d0a03c2c13    registry.k8s.io/etcd:3.5.6-0                       io.containerd.runc.v2    
b5c7d9751f4fdb1dcad44290478f4a8cb9071a9f71973708876d0b1d741b01e0    registry.k8s.io/kube-scheduler:v1.26.0             io.containerd.runc.v2    
cba6ca2986b2673f1ad35213cf5954d3839d2d8d525874c0841307a37f35aa32    registry.k8s.io/pause:3.8                          io.containerd.runc.v2    
d5acf54402096da04955e9c29951253c2f5002e7d66a05dbf2e749075544f7db    registry.k8s.io/pause:3.8                          io.containerd.runc.v2    
e840b846c18388ec64c1dc0f7e1a6b9b94acfa656d82b7a62abef93e6f823073    registry.k8s.io/pause:3.8                          io.containerd.runc.v2    
f53fc660d631dd658b56a10f98c7cb1839b0809010880be3b42232b45ce9e1e5    registry.k8s.io/kube-proxy:v1.26.0                 io.containerd.runc.v2    

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#override-pause-image-containerd

sandbox_image = "registry.k8s.io/pause:3.8"

[spowelljr]

Ah, thanks for the heads up, we need to update the kubeadm constants on master and then I'll delete the existing preload and regenerate it.

[spowelljr]

Here's the kubeadm constants bump PR

#15497

[spowelljr]

PR merged, deleted v1.22.17, v1.23.15, v1.24.9, v1.25.5, and v1.26.0 preloads

Running preload generation now

[spowelljr]

Preloads are done, but this is still going to cause an issue because it's going to manually pull the old image. I think the best way would be to host the kubeadm constants file on the web and just use the local version as a fallback if it can't pull it. But that's only if we care about supporting Kubernetes versions newer than minikube's listed latest supported verison.

[afbjorklund]

Ultimately the best would be if kubeadm could provide the versions, so that it would be directly available after each release

Like how it currently have URLs available for pulling the latest version, etc

Provide some JSON or something, that could be downloaded (and cached)

[afbjorklund]

I guess that 1.26 is fixed, but it will soon start breaking 1.27 pre-releases instead

[Pictor13]

Hello,
I'm getting issue getting the cache for pause image, on Minikube 1.27

❌ Unable to load cached images: loading cached images: stat /home/ipellegrini/.minikube/cache/images/amd64/registry.k8s.io/pause_test2: no such file or directory

Is this the tracking issue related to that problem?

[afbjorklund]

You need to update minikube, to a version including your k8s version.

[spowelljr]

@Pictor13 Your issue was fixed with #16925, like @afbjorklund said update your minikube version and you'll get the fix.

[Pictor13]

Updating solved the problem, thanks for your work guys 🙏