services running on minikube cluster are inaccessible #13788

alexkim-avant · 2022-03-15T17:42:34Z

What Happened?

Hello,

I am running into an issue where after deploying my kubernetes application and service to the minikube cluster, I cannot access them through the command minikube service <service-name>.

Versions:
Docker client 20.10.13
Docker Desktop 4.6.0
minikube v1.25.2
skaffold v1.36.1
macOS Catalina v10.15.7

Steps:
Run minikube start --driver=docker --kubernetes-version=v1.21.10

😄  minikube v1.25.2 on Darwin 10.15.7
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=7911MB) ...
🐳  Preparing Kubernetes v1.21.10 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Run skaffold dev with the following service.yaml file:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: NodePort
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
      name: web
  selector:
    app: my-app

Run minikube service my-service:

🏃  Starting tunnel for service my-service.
🎉  Opening service default/my-service in default browser...
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

This step opens my browser and attempts to connect to the URL http://192.168.49.2:30897/. However, this loads forever and doesn't actually connect me to my app.

Run minikube service list while tunnel is open:

|-------------|------------------------|--------------|-----|
|  NAMESPACE  |          NAME          | TARGET PORT  | URL |
|-------------|------------------------|--------------|-----|
| default     | my-service             | web/9000     |     |
| default     | kubernetes             | No node port |
| default     | postgres-postgresql    | No node port |
| default     | postgres-postgresql-hl | No node port |
| kube-system | kube-dns               | No node port |
|-------------|------------------------|--------------|-----|

shows that there is no URL.

Run docker port minikube while the tunnel is open:

32443/tcp -> 127.0.0.1:65196
5000/tcp -> 127.0.0.1:65197
8443/tcp -> 127.0.0.1:65198
22/tcp -> 127.0.0.1:65194
2376/tcp -> 127.0.0.1:65195

The application port 30897 isn't listed as an accessible port for the docker container running minikube.

If I SSH into the minikube container with minikube ssh and use curl to hit my service, I get a response:

docker@minikube:~$ curl http://192.168.49.2:30897/ping
"pong"
docker@minikube:~$ curl http://localhost:30897/ping
"pong"

So in summary, my service is accessible within the minikube docker container, but running minikube service my-service does not actually connect me to my service.

Attach the log file

log.txt

Operating System

macOS (Default)

Driver

Docker

The text was updated successfully, but these errors were encountered:

zhan9san · 2022-03-16T01:34:11Z

Hi @alexkim-avant

It seems both UI and document need to be improved. I am glad to work on it.

The bridge network in mac is different from in Linux, #7332 (comment)

Here is a workaround for your case

The tunnel is created between your Mac and Cluster IP instead of Node IP.

Get the service cluster IP

It's 10.101.93.174.

$. kubectl get svc -o wide
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE   SELECTOR
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          18m   <none>
web          NodePort    10.101.93.174   <none>        8080:32662/TCP   16m   app=web

Get the tunnel port on your Mac

It's 54725(which is before cluster IP).

$ ps -ef | grep ssh
1195290332 75035 75010   0  9:18AM ttys010    0:00.01 ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N [email protected] -p 53921 -i /Users/xxx/.minikube/machines/minikube/id_rsa -L 54725:10.101.93.174:8080

Access it from localhost

$ curl http://127.0.0.1:54725
Hello, world!
Version: 1.0.0
Hostname: web-746c8679d4-pw4gh

BTW, if you are using ingress, please refer to #12089

alexkim-avant · 2022-03-16T16:06:11Z

Hi @zhan9san,

Thanks for the response. A few follow up questions:

Does the service type need to be NodePort if we're tunneling to the ClusterIP?
Do I still need to run minikube service <service-name> to connect to the service?

zhan9san · 2022-03-16T16:33:38Z

Does the service type need to be NodePort if we're tunneling to the ClusterIP?

Technically, no need.

I think it is a bug. IMHO, if the service type is NodePort, the tunnel port should be .spec.nodePort, but currently, the tunnel port is .spec.targetPort. I'll try to fix it later, if it is verified.

Keep in mind, publishing-services-service-types

NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting :.

Do I still need to run minikube service to connect to the service?

Yes. As I mentioned before, due to the limited bridge network on MacOS, a tunnel is required to access the service inside Kubernetes cluster. And the tunnel is created by minikube service <service-name> --url

If the --url is missing, this command will try to open default web browser to access http://<NodeIP>:<NodePort> if the service type is NodePort.

In short, once the culsterIP type service is created, you can run minikube service <service-name> --url to create a tunnel, and this command will hang until you enter ctrl + c

The available URL would be http://127.0.0.1:<PORT>, where the PORT can be found by ps -ef | grep ssh.

I'll submit another PR to output this URL(http://127.0.0.1:)

zhan9san · 2022-03-16T16:50:37Z

I just noticed the output issue has been fixed in codebase.

❯ ./minikube service web
|-----------|------|-------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT |            URL            |
|-----------|------|-------------|---------------------------|
| default   | web  |        8080 | http://192.168.49.2:30076 |
|-----------|------|-------------|---------------------------|
🏃  Starting tunnel for service web.
|-----------|------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT |          URL           |
|-----------|------|-------------|------------------------|
| default   | web  |             | http://127.0.0.1:65455 |
|-----------|------|-------------|------------------------|
🎉  Opening service default/web in default browser...
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

alexkim-avant · 2022-03-16T17:03:27Z

I have minikube v1.25.2 which was the most recent version when I installed it two days ago. Should I try and reinstall?

zhan9san · 2022-03-17T00:55:04Z

No. The change was only merged into master branch, and it may be in next release. Certainly, you can compile it by yourself.

BTW, the change only fixes the UI issue, showing some friendly output. You can get the output following what I said before.

ArangoGutierrez · 2022-03-24T13:01:22Z

Same here, this is a hot fix

kubectl -n kube-system rollout restart deployment coredns

after that services can discover and be accessed.

spowelljr · 2022-03-24T20:23:23Z

Hi @alexkim-avant, as @zhan9san pointed out that issue was resolved with #13756. You can download the binary with the fix from this comment (#13736 (comment)) if you want to try it and see if it resolves you issue, thanks!

ArangoGutierrez · 2022-03-25T12:34:58Z

I have tried master branch, and can say it works now. no need to restart coredns to get services running

minikube version: v1.25.2
commit: 2b6319258bba0d295bd6bd0d4900fe1646c8be9d

alexkim-avant · 2022-03-25T15:19:42Z

Thank you! I will close the issue now.

ArangoGutierrez · 2022-04-13T16:08:21Z

This is back broken
/reopen

ArangoGutierrez · 2022-04-13T16:08:33Z

/open

k8s-ci-robot · 2022-04-13T16:08:37Z

@ArangoGutierrez: Reopened this issue.

In response to this:

This is back broken
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ArangoGutierrez · 2022-04-13T16:09:00Z

on current master, I had to restart the dns to fix this

spencer-nicol · 2022-05-09T21:31:10Z

I'm running minikube v1.25.2 with Docker v4.8.1 and the ingress addon is not accessible. I see this and other issues related to it have been raised for about 2 years. Is there any movement on this for the docker driver?

zhan9san · 2022-05-10T02:57:21Z

Hi @spencer-nicol

Could you please verify whether #13806 address this issue?

spencer-nicol · 2022-05-10T16:11:21Z

Hi @spencer-nicol

Could you please verify whether #13806 address this issue?

Using the tunnel does allow me to connect to the services, alternatively HyperKit also allows me to do this without tunneling. Is this fundamentally a Docker, MacOS, or virtualization issue? I'm curious if there will eventually be native support?

zhan9san · 2022-05-11T02:14:58Z

Is this fundamentally a Docker, MacOS, or virtualization issue?

It's Docker and MacOS issue. #7332 (comment) and per-container-ip-addressing-is-not-possible

I'm curious if there will eventually be native support?

I don't know, but probably won't.

bugoverfl0w · 2022-05-13T08:07:37Z

kubectl -n kube-system rollout restart deployment coredns

Thanks, it works for me. I don't know why :D

repeating · 2022-07-16T12:51:41Z

Type the following and you'll get a url
minikube service <service-name> --url
For example:

You can access your service through that url

singlamohit1 · 2024-07-26T17:30:47Z

Although using minikube service <service-name> --url , I am able to open the service, but the service opened (mongo-express in my case) is not able to access the mongo db (other service which I created). It is asking me creds even though I mentioned it in the env.
Kindly help

Terkea · 2024-08-04T18:32:37Z

the issue is still present, any fix for this? I cannot use tunnels for what I need

zhan9san mentioned this issue Mar 17, 2022

Update accessing apps service doc #13806

Merged

spowelljr added the kind/support Categorizes issue or PR as a support question. label Mar 24, 2022

alexkim-avant closed this as completed Mar 25, 2022

k8s-ci-robot reopened this Apr 13, 2022

sinevitch mentioned this issue Apr 25, 2022

The terminal needs to be open to run it (mac M1) #14039

Closed

asungur mentioned this issue Apr 25, 2022

add clear documentation that service command on MacOs/Windows needs active terminal seassion #9016

Closed

spowelljr added the long-term-support Long-term support issues that can't be fixed in code label May 4, 2022

spowelljr closed this as completed in #13806 May 17, 2022

Findaa mentioned this issue May 25, 2022

Support Ingress on MacOS, driver docker #12089

Merged

hiroaki2020 mentioned this issue Jun 18, 2022

[Question]My app works properly on docker compose but not on minikube #14326

Closed

jgehrcke mentioned this issue Feb 16, 2023

makefile: minikube-conbench-url during conbench-on-minikube fails on darwin conbench/conbench#742

Closed

lcardito mentioned this issue Aug 21, 2024

[Maven Quickstarts]: hello-world curl request fails eclipse-jkube/jkube#3327

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

services running on minikube cluster are inaccessible #13788

services running on minikube cluster are inaccessible #13788

alexkim-avant commented Mar 15, 2022 •

edited

Loading

zhan9san commented Mar 16, 2022

alexkim-avant commented Mar 16, 2022

zhan9san commented Mar 16, 2022

zhan9san commented Mar 16, 2022

alexkim-avant commented Mar 16, 2022 •

edited

Loading

zhan9san commented Mar 17, 2022

ArangoGutierrez commented Mar 24, 2022

spowelljr commented Mar 24, 2022

ArangoGutierrez commented Mar 25, 2022

alexkim-avant commented Mar 25, 2022

ArangoGutierrez commented Apr 13, 2022

ArangoGutierrez commented Apr 13, 2022

k8s-ci-robot commented Apr 13, 2022

ArangoGutierrez commented Apr 13, 2022

spencer-nicol commented May 9, 2022

zhan9san commented May 10, 2022

spencer-nicol commented May 10, 2022

zhan9san commented May 11, 2022

bugoverfl0w commented May 13, 2022

repeating commented Jul 16, 2022

singlamohit1 commented Jul 26, 2024

Terkea commented Aug 4, 2024

services running on minikube cluster are inaccessible #13788

services running on minikube cluster are inaccessible #13788

Comments

alexkim-avant commented Mar 15, 2022 • edited Loading

What Happened?

Attach the log file

Operating System

Driver

zhan9san commented Mar 16, 2022

alexkim-avant commented Mar 16, 2022

zhan9san commented Mar 16, 2022

zhan9san commented Mar 16, 2022

alexkim-avant commented Mar 16, 2022 • edited Loading

zhan9san commented Mar 17, 2022

ArangoGutierrez commented Mar 24, 2022

spowelljr commented Mar 24, 2022

ArangoGutierrez commented Mar 25, 2022

alexkim-avant commented Mar 25, 2022

ArangoGutierrez commented Apr 13, 2022

ArangoGutierrez commented Apr 13, 2022

k8s-ci-robot commented Apr 13, 2022

ArangoGutierrez commented Apr 13, 2022

spencer-nicol commented May 9, 2022

zhan9san commented May 10, 2022

spencer-nicol commented May 10, 2022

zhan9san commented May 11, 2022

bugoverfl0w commented May 13, 2022

repeating commented Jul 16, 2022

singlamohit1 commented Jul 26, 2024

Terkea commented Aug 4, 2024

alexkim-avant commented Mar 15, 2022 •

edited

Loading

alexkim-avant commented Mar 16, 2022 •

edited

Loading