Minikube commands fail with error Exiting due to DRV_CP_ENDPOINT: Unable to get forwarded endpoint: failed to lookup ip for ""

[munir0b0t]

What Happened?

I am following the tutorial in https://kubernetes.io/docs/tutorials/hello-minikube/ on Archlinux laptop with docker. I did minikube start and the docker container has started up. But when I run minikube dashboard, I get the error

❌  Exiting due to DRV_CP_ENDPOINT: Unable to get forwarded endpoint: failed to lookup ip for ""
💡  Suggestion: 

    Recreate the cluster by running:
    minikube delete <no value>
    minikube start <no value>

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│    Please also attach the following file to the GitHub issue:                             │
│    - /tmp/minikube_dashboard_e82743e0f55314733604a6150dcacabfbc842639_0.log               │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

Deleting and restarting did not work, even after I restarted my entire machine.
When I run minikube logs --file=logs.txt, I get the same error. I have attached all the log files below.

Versions

• Linux: 6.5.7
• Docker: 24.0.6
• minikube: 1.31.2

I'm not entirely sure what the issue is here since the only other reference I found to the error is on Windows. I can verify that the minikube container is running and I am able to log in to the container with docker exec -it minikube bash while the container is running.

Attach the log file

logs.txt
minikube_dashboard_e82743e0f55314733604a6150dcacabfbc842639_0.log
minikube_logs_273c584afb8f56625d862285474a6a21019a3238_0.log

Operating System

Other

Driver

Docker

[pnasrat]

I tried to reproduce but have been unable to do so. I setup a VM with arch linux from Arch-Linux-x86_64-cloudimg-20231001.182377.qcow2, installed docker using pacman, started the service (and added my user to docker group) then downloaded and started minikube and did not see the DRV_CP_ENDPOINT error when I ran minikube dashboard.

The error is raised in ControlPlaneEndpoint but what is notable from your error message is that the hostname printed by the error message is "", ie an empty string. If net.LookupIP is called with "" it will error. Hostname is set by checking the environment for DOCKER_HOST

@munir0b0t on your Archlinux laptop can you tell me what env | grep DOCKER_HOST shows.

[pnasrat]

Looking at the code, there may be some valid values of DOCKER_HOST that the minikube host parsing may fail at.

minikube/pkg/minikube/driver/endpoint.go

Line 41 in bca8883

return hostname, nil, port, fmt.Errorf("failed to lookup ip for %q", hostname)

		hostname := oci.DaemonHost(driverName)

		ips, err := net.LookupIP(hostname)
		if err != nil || len(ips) == 0 {
			return hostname, nil, port, fmt.Errorf("failed to lookup ip for %q", hostname)
		}

in oci.DaemonHost

	if driver == Docker {
		if dh := os.Getenv(constants.DockerHostEnv); dh != "" {
			if u, err := url.Parse(dh); err == nil {
				if u.Host != "" {
					return u.Hostname()
				}
			}
		}
	}
	return DefaultBindIPV4

https://github.com/kubernetes/minikube/blob/master/pkg/drivers/kic/oci/oci.go#L701

A simple reproducer using "tcp://:" (that docker has as valid) shows a failure where we get a host but an empty hostname from the parsed URL

https://go.dev/play/p/l_rSOLHq0TW

u.Host: ":"
u.Hostname() ""

[pnasrat]

Indeed with running minikube start with DOCKER_HOST="tcp://:" exported and docker configured for localhost access per https://docs.docker.com/config/daemon/remote-access/#configuring-remote-access-with-systemd-unit-file I get theDRV_CP_ENDPOINT error

❌  Exiting due to DRV_CP_ENDPOINT: Unable to get forwarded endpoint: failed to lookup ip for ""

[pnasrat]

/kind bug

[pnasrat]

/assign

[munir0b0t]

@munir0b0t on your Archlinux laptop can you tell me what env | grep DOCKER_HOST shows.

Seems like you've found the issue but there is no such variable defined in my environment.

The start command for docker in my systemd unit file is ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock.

[pnasrat]

Hmm @munir0b0t I'd still like to understand what it is about your config that is triggering the issue. Could you also :

env | grep CONTAINER_HOST
docker context ls

[munir0b0t]

$ env | grep CONTAINER_HOST
$ docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT               ERROR
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock   
$ docker context inspect 
[
    {
        "Name": "default",
        "Metadata": {},
        "Endpoints": {
            "docker": {
                "Host": "unix:///var/run/docker.sock",
                "SkipTLSVerify": false
            }
        },
        "TLSMaterial": {},
        "Storage": {
            "MetadataPath": "\u003cIN MEMORY\u003e",
            "TLSPath": "\u003cIN MEMORY\u003e"
        }
    }
]

[munir0b0t]

I did some more digging into my problem and I noticed that when I run minikube start, the generated ~/.minikube/profiles/minikube/config.json has empty strings as the node IP. As per the code in https://github.com/kubernetes/minikube/blob/master/pkg/minikube/driver/endpoint.go#L60, this will throw the error I'm seeing. It tried setting the value to 127.0.0.1, but then minikube dashboard just hangs.

Now I guess the question is why is the node IP empty. Perhaps this can help with debugging the issue.

[pnasrat]

@munir0b0t hmm - so the way the docker driver works is there is a minikube docker container that within it is also running containers. For the docker driver you can get what the node IP should be via docker inspect. Note that setting it to 127.0.0.1 won't work as that's not where the mapped port to the container through to the apiserver is (for comparison curl -I -k https://127.0.0.1:8443 would fail but curl -I -k https://192.168.49.2:8443 responds from the host machine.

Can you do the below can you try starting minikube after a purge delete and see if the config json still has no ip set in the node

minikube stop
minikube delete --purge --all
minikube start

docker inspect \
  -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' minikube
  192.168.49.2

If that doesn't return an IP address please attach the output of

docker network inspect minikube

For comparison on my system it matches here jq '.["Nodes"]' ~/.minikube/profiles/minikube/config.json

[
  {
    "Name": "",
    "IP": "192.168.49.2",
    "Port": 8443,
    "KubernetesVersion": "v1.28.3",
    "ContainerRuntime": "docker",
    "ControlPlane": true,
    "Worker": true
  }
]

[munir0b0t]

The first command did return an IP, but it's not in config.json file. I include the full output of docker inspect also. Perhaps there's something different there between your system and mine.

$ docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' minikube
192.168.49.2


$ jq '.["Nodes"]' ~/.minikube/profiles/minikube/config.json
[
  {
    "Name": "",
    "IP": "",
    "Port": 8443,
    "KubernetesVersion": "v1.27.4",
    "ContainerRuntime": "docker",
    "ControlPlane": true,
    "Worker": true
  }
]

$ docker network inspect minikube 
[
    {
        "Name": "minikube",
        "Id": "a79beaa375aeaa3d072432de0cc8fcbdf4f451ecea18ff716db1990ce4171f8c",
        "Created": "2023-11-07T13:41:38.35840072-08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.49.0/24",
                    "Gateway": "192.168.49.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "52ee1136893c36fa543121206859e7c2030f4ce87492f9525c3c6e281b158c48": {
                "Name": "minikube",
                "EndpointID": "c4851ce5f1c7dd4248f65d1e5ef77abb79c13e99a6bb6de5031c9c1b14364651",
                "MacAddress": "02:42:c0:a8:31:02",
                "IPv4Address": "192.168.49.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "--icc": "",
            "--ip-masq": "",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {
            "created_by.minikube.sigs.k8s.io": "true",
            "name.minikube.sigs.k8s.io": "minikube"
        }
    }
]

[munir0b0t]

I think I figured out the issue. I have systemd network file for bridge devices that is overriding the configuration of the bridge device created by minikube.

$ cat /etc/systemd/network/60-bridge.network
[Match]
Name=br*

[Network]
DHCP=yes

The network bridge device created by minikube matches this and causes a conflict with the network confguration. After changing the pattern to br0, everything works as expected and I'm able to spin up the dashboard.

Thanks for helping me debug this. I'd say that this is mostly a configuration issue and not a bug in minikube. But this should help anyone else who might run into a similar issue.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.