Improvements for using x86 binary on arm64 machine

[spowelljr]

Related #19110 #18895 #12274

Improvement 1

Issue:
If a user desired to run an x86 cluster on an arm64 machine, they would use an x86 minikube binary. However, due to our logic we would use an x86 ISO, but we would then download arm64 preload and K8s binaries, resulting in arch mismatches between the cluster and the K8s binaries, resulting in minikube failing to start.

Fix:
We now download the preloads and K8s binary strictly based on the minikube binary arch, resulting in the proper K8s binaries being downloaded.

Improvement 2

Issue:
When starting minikube using an x86 binary on an arm64 machine and using the Docker driver, we would incorrectly download both the x86 and arm64 kicbase images, and use the arm64 image for running the container. This was due to running a docker pull without the --platform flag, using the host machines default arch, being arm64.

Fix:
We now specify the --platform flag on the docker pull, resulting in correctly only pulling the x86 kicbase image.

Improvement 3

Issue:
After resolving the above two issues I was able to attempt to to start an x86 kicbase with x86 K8s binaries on an arm64 machine, this resulted in the failure could not detect clock speed from output from kubelet. And as per kubernetes-sigs/kind#3621 (comment), Docker's "cross platform images" only use partial emulation that cannot run a workload like Kubernetes.

Fix:
Hard fail if a user attempts to start an x86 cluster on arm64 with the Docker driver.

$ minikube-darwin-amd64 start --driver docker
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                          │
│    You are trying to run the amd64 binary on an M1 system.                                               │
│    Please consider running the darwin/arm64 binary instead.                                              │
│    Download at https://github.com/kubernetes/minikube/releases/download/v1.33.1/minikube-darwin-arm64    │
│                                                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────╯
😄  minikube v1.33.1 on Darwin 14.5
✨  Using the docker driver based on user configuration

💣  Exiting due to PROVIDER_DOCKER_INCORRECT_ARCH: Cannot use amd64 minikube binary to start minikube cluster with Docker driver on arm64 machine
💡  Suggestion: Download and use arm64 version of the minikube binary
📘  Documentation: https://minikube.sigs.k8s.io/docs/start/

Improvement 4

Issue:
Users have run into an issue where they try starting minikube with the x86 binary, which downloads the x86 kicbase, they then download and use the arm64 binary, but due to kicbase having identical tags uses the existing x86 version but then uses arm64 preload and K8s binaries, resulting in arch mismatches, and causing minikube to fail to start.

Fix:
Our previous check just did docker images and then checked for the existence of the expected tag. Improved the check to ensure the image matches the expected arch, if it doesn't it downloads the correct one.

[spowelljr]

/ok-to-test

[minikube-pr-bot]

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Hyperkit_macOS	TestMountStart/serial/VerifyMountSecond (gopogh)	0.00 (chart)
Docker_Linux_docker_arm64	TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh)	8.16 (chart)

To see the flake rates of all tests by environment, click here.

[spowelljr]

/ok-to-test

[minikube-pr-bot]

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19205) |
+----------------+----------+---------------------+
| minikube start | 50.4s    | 49.3s               |
| enable ingress | 25.4s    | 25.7s               |
+----------------+----------+---------------------+

Times for minikube start: 50.5s 49.4s 52.2s 49.9s 50.0s
Times for minikube (PR 19205) start: 47.6s 51.1s 47.7s 50.3s 50.0s

Times for minikube ingress: 25.4s 24.0s 25.0s 27.5s 25.0s
Times for minikube (PR 19205) ingress: 27.5s 23.5s 24.0s 27.5s 25.9s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19205) |
+----------------+----------+---------------------+
| minikube start | 23.5s    | 23.7s               |
| enable ingress | 22.0s    | 21.9s               |
+----------------+----------+---------------------+

Times for minikube start: 25.5s 21.3s 24.3s 22.4s 24.1s
Times for minikube (PR 19205) start: 22.0s 24.9s 24.5s 24.5s 22.7s

Times for minikube ingress: 21.3s 21.8s 22.8s 21.8s 22.3s
Times for minikube (PR 19205) ingress: 22.8s 21.8s 21.8s 21.3s 21.8s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19205) |
+----------------+----------+---------------------+
| minikube start | 22.0s    | 21.4s               |
| enable ingress | 45.9s    | 38.3s               |
+----------------+----------+---------------------+

Times for minikube (PR 19205) ingress: 48.3s 31.8s 47.8s 31.8s 31.8s
Times for minikube ingress: 47.8s 46.8s 47.8s 47.8s 39.3s

Times for minikube start: 23.5s 19.9s 22.8s 20.7s 23.3s
Times for minikube (PR 19205) start: 23.8s 20.3s 20.2s 22.6s 19.9s

[medyagh]

one nit and then it will be good to go.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, spowelljr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [medyagh,spowelljr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment