Whats the priority: /var/lib/rancher/rke2/agent/images/ or registries.yaml

[rogersollberger]

Hi,

in the tarball method, i can copy all images to the /var/lib/rancher/rke2/agent/images/ folder and they get loaded when the rke2 service restarts. After that i copied a deployment manifest to the manifest folder which uses the image i copied to the images folder.

everything works fine.

now for this image, there is now a new version available. so i thought i copy the new image to the images folder (/var/lib/rancher/rke2/agent/images/) and also and updated manifest to the manifest folder and then restart the service.

my expectation was it will then use the new image. but he cannot pull this image now. and i wonder why?
Could it be, there is a registreis.yaml on the service in the meanwhile?

Is there a priority order which one rke2 uses first or an exclusion? i mean if a registries.yaml is in place images/ folder gets ignored.

Could someone help me in this behavior?
Thanks a lot.

[Martin-Weiss]

crictl images <- can you see the new image, there before you update image in the deployment?
What is the image pull policy?

I guess that the problem might be a race condition between updating the manifest with the new image and this gets applied within 1 minute and this happens even before the rke2 restart happend to add the image to the containerd local image repository.

[rogersollberger]

I did now a test. I removed the registries.yaml and then the deployment worked fine so it could pull the images correctly. Is use
imagePullPolicy: IfNotPresent normally.

So with this tests it seems, that if the registries.yaml is in charge, he does no longer load images from the images/ folder.
So technically then we have to move the images into the private registry.

[Martin-Weiss]

I guess the update of the image tag in the deployment has happened before the rke2 restart happend. (adjusting the yaml in the manifests folder will apply the change within a minute).

The "rule" is that if the image is in the containerd store with the exact same tag - the "IfNotPresent" will ensure it will not download the image once more - so the registries.yaml does not come into the game at all IF the image is in the container repo, already. (verify with crictl images)

[brandond]

The image tarballs are always imported into containerd. There is no interaction between the image import and registries.yaml. Having the image imported provides the same result as manually pulling the image from a repository, and it does not prevent the kubelet from checking the repository for updates if your image pull policy is set to Always or you use the latest tag.