Failed to running minikube under lima

[thr3a]

What Happened?

I have tried to install minikube on a virtual environment built using lima but it fails.

Environment

Macbook M1

❯ lima -v
limactl version 0.11.3
❯ which lima
/opt/homebrew/bin/lima

❯ docker -v
Docker version 20.10.17, build 100c70180f
❯ which docker
/opt/homebrew/bin/docker
❯ docker context list
NAME            DESCRIPTION                               DOCKER ENDPOINT                                      KUBERNETES ENDPOINT                ORCHESTRATOR
default         Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                          https://127.0.0.1:6443 (default)   swarm
lima-docker *                                             unix:///Users/thr3a/.lima/default/sock/docker.sock

❯ minikube version
minikube version: v1.26.1
commit: 62e108c3dfdec8029a890ad6d8ef96b6461426dc
❯ which minikube
/opt/homebrew/bin/minikube

Command

minikube start --driver=docker --container-runtime=containerd --alsologtostderr -v=1

I confirmed that the installation failed as well after running the initialization command (minikube delete --all --purge).

Attach the log file

log.txt

Operating System

macOS (Default)

Driver

Docker

[afbjorklund]

You might need to set the DOCKER_HOST, explicitly. I don't think all clients understand the "context" (yet)

But the root cause seems to be stray binary characters in the output from the commands over ssh ?

[afbjorklund]

We discussed various ways to run under lima, but it is not yet a supported docker platform

• Running minikube under lima #12926

The default is to just install containerd and kubeadm on the Lima VM directly... (k8s.yaml)

It should support running with docker-in-docker too, even rootless, but it is not tested

[afbjorklund]

It worked ok with lima on Linux, using rootless docker

So the docker host (vs context) warnings were not related

[thr3a]

I have tried several patterns and have successfully installed minikube only when the virtual machine architecture is aarch64 and rootful.

In the case of x86_64 & rootful, the installation proceeded better than in the other cases but eventually failed.

lima arch \ rootless?	rootful	rootless(default)
aarch64	⭕	❌
x86_64	❌	❌

x86_64-rootful.txt

※ In both cases, DOCKER_HOST is set.

[afbjorklund]

Could be related to the flaky "user" networking on Mac then (known lima issue...) Can you try with "vde", to see if it helps ?

[afbjorklund]

It seems to have the same issue as on other platforms, when the Unix socket is not local but tunneled to another machine in secret... It seems to be using 192.168.52.2 for the IP, while Lima wants it to be 127.0.0.1 (tunneled)

• minikube start fails in docker-from-docker context #13950

In the Lima k3s/k8s examples, this is done with sed on kube config...

Here it is even worse, since it is tunneled again (to the fakenode)

[afbjorklund]

It might have worked better with lima minikube, but then you can't use the local host editor and browser properly (similar to the local "none" vs the remote "ssh" driver issues)

[thr3a]

@afbjorklund
Sorry, I don't know much about vde, do you mean vde_vmnet?

If I install vde_vmnet and add the following network settings, will the 'vde' configuration requirements be met?

networks:
  - lima: host

[afbjorklund]

Turns out the 192.168.58 was a freak show with minikube deciding that 192.168.49 was in use on the host, and therefore deciding to use 192.168.49 + 9 for the docker network without noticing it was the same network as the Lima VM...

Anyway, minikube doesn't work with lima and tunneling sockets. It expects real IPs and TCP sockets, the way it was with docker-machine

For now, you have to run lima minikube instead. (Linux)

It could be made to work, if there was any interest. But so far, there hasn't been - not from lima, not from kind. So we're doing the "qemu" driver in minikube instead (of lima)

[thr3a]

I realized that the way I tried to run minikube under lima was not the standard way.
Actually, I also tried lima minikube(arch: x86_64), but it was too slow, so I decided to install minikube on Linux host.
(I understand that the slow problem is not caused by minikube or lima.)

Thank you.

[afbjorklund]

I realized that the way I tried to run minikube under lima was not the standard way.

It is quite normal, if you try to replace Docker Desktop with Lima (docker).

Just not working very well