Cannot pull images from private registry for 1 out of 3 worker nodes

[tetraal]

Hi,

I have a development RKE2 cluster that consists of 1 master node and 3 worker nodes running RKE2 v1.26.0 that has been running for months. I am pulling images from a private registry and have configured the registries.yaml file correctly as I can successfully pull from 2 of the 3 worker nodes successfully. On the one failing node it fails to pull the images and returns a '401 Unauthorized' for any of the images from the private registry. However, on that same failing node I can successfully pull the images using 'crictl'

Looking for any suggestions/thoughts or directions.

Thanks.

[brandond]

can't really suggest anything other than comparing the regiestries.yaml between the working/non-working nodes, and looking at the containerd.log to see what's going on. I suspect that some of the nodes are failing to pull from the private registry, and are falling back to docker.io or whatever the default endpoint is - and getting a 401 from that.

[tetraal]

Thanks for the response and I should have mentioned this in the first place. I did do the registries.yaml comparison and even copied one from the working one to the failing node and restarted and I don't see anything in containerd.log that seems out of synch. This image is not published anywhere except in this private registry.

So yesterday I did some more testing and if I add an imagepullsecrets field with a secret using the same username/password it pulls successfully on that one failing node. If I use the same K8S manifest file, remove the imagepullsecrets field and publish it to one of the other working nodes it pulls successfully without the imagepullsecrets field. Again, on the failed node I can pull the images successfully if i first use crictl to pull the image and then deploy the k8s manifest file without the imagepullsecrets field the pod starts up.

One question I do have, is there a way to debug containerd from within rke2. I tried setting 'debug' in the config.toml after but that doesn't work because RKE2 at startup regenerates the config.toml file.

The failing node is behaving the same was as if I tried to pull the image using the 'ctr' command without specifying a username/password.