Containerd image labels are lost after restarting k3s

[nouzun]

Environmental Info:
k3s version v1.31.2+k3s1 (6da2042)
go version go1.22.8

Node(s) CPU architecture, OS, and Version:
x86_64 GNU/Linux

Cluster Configuration:
Single node

Describe the bug:
I import images via air-gap image tar files. They are pinned in containerd to ensure that they remain available and are not pruned by Kubelet garbage collection.

After restarting k3s, the containerd labels are lost.

Steps To Reproduce:

I store the air-gap image tar file under /var/lib/rancher/k3s/agent/images and start k3s, k3s labels the images using io.cattle.k3s.pinned=pinned and io.cri-containerd.pinned=pinned.

ctr images list:

docker.io/rancher/klipper-helm:v0.9.3-build20241008                                                application/vnd.docker.distribution.manifest.v2+json sha256:39a2219d2d868151120f51e0838f762eb8c60fe57fa9a8ada3682736c33dc982 181.7 MiB linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
docker.io/rancher/klipper-lb:v0.4.9                                                                application/vnd.docker.distribution.manifest.v2+json sha256:44b0d0278f9a93628c4f0ef99f9192ab3a469febd9b056c01beff16fa2c11afe 12.1 MiB  linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
docker.io/rancher/local-path-provisioner:v0.0.30                                                   application/vnd.docker.distribution.manifest.v2+json sha256:3665fcdda66135216ceaadbc23a10268c50dc4f389ded7fa34b77af02f2e0523 49.6 MiB  linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
docker.io/rancher/mirrored-coredns-coredns:1.11.3                                                  application/vnd.docker.distribution.manifest.v2+json sha256:cdf67db06aea868d7bb8a5ffbd889e082fde6937f17022c4474bfcf7c4851833 60.3 MiB  linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
docker.io/rancher/mirrored-library-busybox:1.36.1                                                  application/vnd.docker.distribution.manifest.v2+json sha256:55c5e6244e8b1a1d0d4068e6670890de18e30dc28e419b925a5f642773d1258d 4.3 MiB   linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
docker.io/rancher/mirrored-library-traefik:2.11.10                                                 application/vnd.docker.distribution.manifest.v2+json sha256:c4b8a75c120586289628b17f65a86187ceeb65a1000dd65d29534a823dc0cd77 160.9 MiB linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
docker.io/rancher/mirrored-metrics-server:v0.7.2                                                   application/vnd.docker.distribution.manifest.v2+json sha256:f32be890c4074b914b210379c6580bc900e594d9d6b438abb23149d306b66293 65.0 MiB  linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
docker.io/rancher/mirrored-pause:3.6                                                               application/vnd.docker.distribution.manifest.v2+json sha256:79b611631c0d19e9a975fb0a8511e5153789b4c26610d1842e9f735c57cc8b13 669.8 KiB linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned

Then I delete the tar file since the images are already imported. After restarting the k3s service via systemctl restart k3s, the labels are lost.

ctr images list:

docker.io/rancher/klipper-helm:v0.9.3-build20241008                                                application/vnd.docker.distribution.manifest.v2+json sha256:39a2219d2d868151120f51e0838f762eb8c60fe57fa9a8ada3682736c33dc982 181.7 MiB linux/amd64 io.cri-containerd.image=managed
docker.io/rancher/klipper-lb:v0.4.9                                                                application/vnd.docker.distribution.manifest.v2+json sha256:44b0d0278f9a93628c4f0ef99f9192ab3a469febd9b056c01beff16fa2c11afe 12.1 MiB  linux/amd64 io.cri-containerd.image=managed
docker.io/rancher/local-path-provisioner:v0.0.30                                                   application/vnd.docker.distribution.manifest.v2+json sha256:3665fcdda66135216ceaadbc23a10268c50dc4f389ded7fa34b77af02f2e0523 49.6 MiB  linux/amd64 io.cri-containerd.image=managed
docker.io/rancher/mirrored-coredns-coredns:1.11.3                                                  application/vnd.docker.distribution.manifest.v2+json sha256:cdf67db06aea868d7bb8a5ffbd889e082fde6937f17022c4474bfcf7c4851833 60.3 MiB  linux/amd64 io.cri-containerd.image=managed
docker.io/rancher/mirrored-library-busybox:1.36.1                                                  application/vnd.docker.distribution.manifest.v2+json sha256:55c5e6244e8b1a1d0d4068e6670890de18e30dc28e419b925a5f642773d1258d 4.3 MiB   linux/amd64 io.cri-containerd.image=managed
docker.io/rancher/mirrored-library-traefik:2.11.10                                                 application/vnd.docker.distribution.manifest.v2+json sha256:c4b8a75c120586289628b17f65a86187ceeb65a1000dd65d29534a823dc0cd77 160.9 MiB linux/amd64 io.cri-containerd.image=managed
docker.io/rancher/mirrored-metrics-server:v0.7.2                                                   application/vnd.docker.distribution.manifest.v2+json sha256:f32be890c4074b914b210379c6580bc900e594d9d6b438abb23149d306b66293 65.0 MiB  linux/amd64 io.cri-containerd.image=managed
docker.io/rancher/mirrored-pause:3.6                                                               application/vnd.docker.distribution.manifest.v2+json sha256:79b611631c0d19e9a975fb0a8511e5153789b4c26610d1842e9f735c57cc8b13 669.8 KiB linux/amd64 io.cri-containerd.image=managed

The same issue happens if I manually label the images:

For example:

ctr images label docker.io/rancher/klipper-lb:v0.4.9 io.cattle.k3s.pinned=pinned io.cri-containerd.pinned=pinned

ctr images list:

docker.io/rancher/klipper-lb:v0.4.9                                                                application/vnd.docker.distribution.manifest.v2+json sha256:44b0d0278f9a93628c4f0ef99f9192ab3a469febd9b056c01beff16fa2c11afe 12.1 MiB  linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned

Restart k3s via systemctl restart k3s and containerd receives ImageUpdate event and updates the labels:

time="2024-11-25T01:06:06.900479317-08:00" level=info msg="ImageUpdate event name:\"docker.io/rancher/klipper-lb:v0.4.9\"  labels:{key:\"io.cri-containerd.image\"  value:\"managed\"}"

BTW I realized that containerd receives the ImageUpdate event after restart only if I label the image with io.cattle.k3s.pinned=pinned, otherwise the images keep their labels.

• Installed K3s:

/usr/local/bin/k3s \
    server \
	'--cluster-init' \
	'--write-kubeconfig-mode=644' \
	'--disable=traefik' \
	'--disable=local-storage' \
	'--secrets-encryption' \
	'--embedded-registry' \
	'--prefer-bundled-bin' 

Expected behavior:
I would expect containerd image labels to be persistent after restarting k3s.

Actual behavior:
The containerd image labels are lost after restarting k3s service.

[brandond]

Then I delete the tar file since the images are already imported.

If you want the images to remain pinned, leave the tarballs in the images dir.

On startup, K3s removes the pinned label from all images in the image store, then re-imports and re-pins everything from the images dir. This is to ensure that "old" images that are no longer present on disk don't remain pinned and take up space after their source archives have been removed from disk.

If this doesn't work for you, then I would suggest setting up a private registry mirror to hold your images, instead of relying on import pinning.

[nouzun]

@brandond I guess k3s removes the images that have io.cattle.k3s.pinned label, right? If i manually pin images with io.cri-containerd.pinned, does k3s remove those images as well on startup?

[brandond]

Yes, it only unpins things that have a label indicating they were pinned by k3s. So if you remove the k3s label, it should leave the pin - assuming you don't have the image archive around any longer to cause it to re-apply the label.

k3s/pkg/agent/containerd/containerd.go

Lines 248 to 264 in 1b7dd76

	// clearLabels removes the pinned labels on all images in the image store that were previously pinned by k3s
	func clearLabels(ctx context.Context, client *containerd.Client) error {
	var errs []error
	imageService := client.ImageService()
	images, err := imageService.List(ctx, fmt.Sprintf("labels.%q==%s", k3sPinnedImageLabelKey, k3sPinnedImageLabelValue))
	if err != nil {
	return err
	}
	for _, image := range images {
	delete(image.Labels, k3sPinnedImageLabelKey)
	delete(image.Labels, labels.PinnedImageLabelKey)
	if _, err := imageService.Update(ctx, image, "labels"); err != nil {
	errs = append(errs, errors.Wrap(err, "failed to delete labels from image "+image.Name))
	}
	}
	return merr.NewErrors(errs...)
	}