Spegel functioning but no cache hits

[craigcabrey]

k3s: v1.29.5+k3s1

I used this trick from another answer to look at spegel stats:

kubectl get --raw /api/v1/nodes/<node>/proxy/metrics | grep -F 'spegel'

With sample output:

libp2p_rcmgr_streams{dir="inbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
libp2p_rcmgr_streams{dir="outbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 1
# HELP spegel_advertised_image_digests Number of image digests advertised to be available.
# TYPE spegel_advertised_image_digests gauge
spegel_advertised_image_digests{registry="docker.io"} 35
spegel_advertised_image_digests{registry="gcr.io"} 1
spegel_advertised_image_digests{registry="ghcr.io"} 18
spegel_advertised_image_digests{registry="quay.io"} 15
spegel_advertised_image_digests{registry="registry-1.docker.io"} 1
spegel_advertised_image_digests{registry="registry.k8s.io"} 5
# HELP spegel_advertised_image_tags Number of image tags advertised to be available.
# TYPE spegel_advertised_image_tags gauge
spegel_advertised_image_tags{registry="docker.io"} 14
spegel_advertised_image_tags{registry="gcr.io"} 1
spegel_advertised_image_tags{registry="ghcr.io"} 3
spegel_advertised_image_tags{registry="quay.io"} 3
spegel_advertised_image_tags{registry="registry.k8s.io"} 1
# HELP spegel_advertised_images Number of images advertised to be available.
# TYPE spegel_advertised_images gauge
spegel_advertised_images{registry="docker.io"} 49
spegel_advertised_images{registry="gcr.io"} 2
spegel_advertised_images{registry="ghcr.io"} 21
spegel_advertised_images{registry="quay.io"} 18
spegel_advertised_images{registry="registry-1.docker.io"} 1
spegel_advertised_images{registry="registry.k8s.io"} 6
# HELP spegel_advertised_keys Number of keys advertised to be available.
# TYPE spegel_advertised_keys gauge
spegel_advertised_keys{registry="docker.io"} 355
spegel_advertised_keys{registry="gcr.io"} 14
spegel_advertised_keys{registry="ghcr.io"} 287
spegel_advertised_keys{registry="quay.io"} 169
spegel_advertised_keys{registry="registry-1.docker.io"} 4
spegel_advertised_keys{registry="registry.k8s.io"} 49

I have the required registries.yaml deployed on every node. Some of my images use latest, but many don't. Furthermore, I use k8s-digester (as recommended by the spegel project itself) to resolve image tags to a digest. This should make for a very cache friendly workload.

Yet, I see no cache hits. My only suspicion is that it's related to my dual stack setup, with IPv6 as a priority (i.e. the first address). Is there some way to access the logs of spegel?

[brandond]

Spegel logs are mixed in with all the rest of the k3s logs in the systemd journal. If you run with --debug you'll get more of them. You can also check the containerd logs.

If you're using ipv6 as the primary address family on your nodes, you're probably running into #9897 (comment)

[craigcabrey]

Finally getting back to this. I don't even see the configs referenced in that issue being generated for me. For example, here is a worker node:

root@ms01-node-1:/var/lib/rancher/k3s/agent/etc/containerd# ls -l certs.d/
total 0
drwx------. 2 root root 24 Aug 23 17:42 '[::1]:6443'
drwx------. 2 root root 24 Aug 23 17:42  _default
root@ms01-node-1:/var/lib/rancher/k3s/agent# journalctl -u k3s-agent | grep toml
[snip]
Aug 16 07:12:41 ms01-node-1 k3s[3064083]: time="2024-08-16T07:12:41Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.tom -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Aug 23 17:42:35 ms01-node-1 k3s[1814567]: time="2024-08-23T17:42:35Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.tom -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"

All of my control plane nodes have the embedded-registry: true field set.

[brandond]

Yes, the issue is going to be related to the IPV6 problem I linked above.