Error downloading kic artifacts

[chrisHLY]

重现问题所需的命令：
minikube start --memory=4096 --cpus=2 --kubernetes-version=v1.30.0

失败的命令的完整输出：

E0716 16:30:50.296015 18021 cache.go:189] Error downloading kic artifacts: failed to download kic base image or any fallback image
🔥 Creating docker container (CPUs=2, Memory=4096MB) ...
🤦 StartHost failed, but will try again: creating host: create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -d /var/lib: exit status 125
stdout:

stderr:
Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e' locally
docker: Error response from daemon: Get "https://gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.

🤷 docker "minikube" 缺失 container，将重新创建。
🔥 Creating docker container (CPUs=2, Memory=4096MB) ...
😿 启动 docker container 失败。运行 "minikube delete" 可能需要修复它： recreate: creating host: create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -d /var/lib: exit status 125
stdout:

stderr:
Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e' locally
docker: Error response from daemon: Get "https://gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.

❌ 因 GUEST_PROVISION 错误而退出：error provisioning guest: Failed to start host: recreate: creating host: create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -d /var/lib: exit status 125
stdout:

stderr:
Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e' locally
docker: Error response from daemon: Get "https://gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.

minikube logs命令的输出：

logs.txt

使用的操作系统版本：
macOs Ventura 13.0.1

[ComradeProgrammer]

@medyagh @spowelljr
I guess the reason is like this:
at pkg/drivers/kic/kic.go:180, oci.PrepareContainerNode was called, before the extraction of preload tarball (which happens at pkg/drivers/kic/kic.go:187, 7 lines after oci.PrepareContainerNode was called)

Under this circumstance:

1. oci.PrepareContainerNode will try to run prepareVolumeSideCar, which will start a kicbase container.
2. But at this time the preload has not been extracted, so docker will try to pull image via gcr.io or dockerhub
3. In China mainland, pulling kicbase via gcr.io or dockerhub will fail due to GFW, so the start process fails

Does this make sense?

BTW I think I was wrong about the reason why preload was not used in China mainland. It seems that it was not proxy. And this seems to be the actual reason why preload never works in China mainland

[spowelljr]

@ComradeProgrammer I don't think this is the issue. oci.PrepareContainerNode creates the minikube and minikube-preload-sidecar. Once that's done it extracts the preloads to the volume created in oci.PrepareContainerNode.

oci.PrepareContainerNode:

docker volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true

docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var docker.io/kicbase/build:v0.0.44-1721064868-19249@sha256:f2789f25c9e51cdeb9cef760e15dc838ef08abd5bb1913311c1eabedda231e8c -d /var/lib

oci.ExtractTarballToVolume:

docker run --rm --entrypoint /usr/bin/tar -v /Users/powellsteven/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.2-docker-overlay2-arm64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir docker.io/kicbase/build:v0.0.44-1721064868-19249@sha256:f2789f25c9e51cdeb9cef760e15dc838ef08abd5bb1913311c1eabedda231e8c -I lz4 -xf /preloaded.tar -C /extractDir

I assume you're referencing the log docker run --rm --name minikube-preload-sidecar, that's called in oci.PrepareContainerNode, not oci.ExtractTarballToVolume. In the logs above the user never even hits oci.ExtractTarballToVolume because because creating the volume with the kicbase fails first.

[ComradeProgrammer]

So accoding to the discussion on Wednesday

• it seems that the reason was indeed caused by failing to pull kicbase, and preload does not include the kicbase image.
• one of the possible way to solve it:
• • to solve it is to provide a tarball of kicbase during the minikube release
• • allow minikube to download and verify the tarball to load the kicbase image from github
    if this is ok I am going to start having a try on this
    @medyagh @spowelljr

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.