Mirrors configured in /etc/rancher/k3s/registries.yaml do not take effect

[Judy166]

Environmental Info:
K3s Version:
k3s version v1.30.5+k3s1 (9b58670)
go version go1.22.6

Node(s) CPU architecture, OS, and Version:

Cluster Configuration:

Describe the bug:

what I did as below
It seems that the mirrors I have configured do not be used

 1.

[root@k3s-node1 k3s]# pwd
/etc/rancher/k3s
[root@k3s-node1 k3s]# cat registries.yaml 
mirrors:
  docker.io:
    endpoint:
      - "https://registry.cn-hangzhou.aliyuncs.com/"

 2.

systemctl restart k3s

 3.

[root@k3s-node1 k3s]# cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml 
# File generated by k3s. DO NOT EDIT.

server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]

[host]
[host."https://registry.cn-hangzhou.aliyuncs.com/v2"]
  capabilities = ["pull", "resolve"]

 4.

[root@k3s-node1 k3s]# kubectl get pods -n ruoyi
NAME                     READY   STATUS             RESTARTS   AGE
mysql-6f876d789c-bgj69   0/1     ImagePullBackOff   0          78m
[root@k3s-node1 k3s]# kubectl describe pod mysql-6f876d789c-bgj69 -n ruoyi | grep Failed
  Warning  Failed          78m                  kubelet            Failed to pull image "mysql:8.2.0": failed to pull and unpack image "docker.io/library/mysql:8.2.0"


 5.

[root@k3s-node1 k3s]# crictl info | grep -A 5 "registry"
    "registry": {
      "configPath": "/var/lib/rancher/k3s/agent/etc/containerd/certs.d",
      "mirrors": null,
      "configs": null,
      "auths": null,
      "headers": null
--
    "sandboxImage": "registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause:3.6",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",

Steps To Reproduce:

• Installed K3s:

Expected behavior:

Actual behavior:

Additional context / logs:

[brandond]

The mirror configuration in containerd is correct - you see your mirror listed in /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml and the containerd configPath is set to /var/lib/rancher/k3s/agent/etc/containerd/certs.d which will reference that file.

Check containerd.log to see why it's failing to pull the image from your mirror. Image pull failure events from the kubelet do NOT show all errors encountered by containerd when pulling, it only captures the terminal failure to pull from the default endpoint.