Access port forwarded via UI from within k8s pod

[adamkpickering]

I'm making this discussion to continue a thread from #3326 that is unrelated to the original topic of that issue. Relevant posts:

@TheGreatBey0nd said:

same behavior, when i try to connect from a pod within the cluster to a forwarded port. Is this a different bug?

@adamkpickering said:

@TheGreatBey0nd it's hard to know without better understanding what you're trying to do. Can you be more specific? If you're trying to connect to the port on the host that has been forwarded to something inside the Rancher Desktop VM from inside a pod, that is definitely not going to work. This is because that port is accessible only on the host, and not inside the Rancher Desktop VM or your pod.

@TheGreatBey0nd said:

@TheGreatBey0nd it's hard to know without better understanding what you're trying to do. Can you be more specific? If you're trying to connect to the port on the host that has been forwarded to something inside the Rancher Desktop VM from inside a pod, that is definitely not going to work. This is because that port is accessible only on the host, and not inside the Rancher Desktop VM or your pod.

that is exactly what I am trying to do -> connect from a pod inside the cluster to an forwarded windows port on RD UI).

But which component restrict the communication?
Why the forwarded port isn´t accessible from a pod inside the cluster? Whats the difference between a "normal" local windows port (e.g. a Tomcat) on my Windows host and a forwarded RD UI port?

Setup:

Windows Host, Rancher Desktop 1.6.2
Ubuntu Pod with Tomcat (Port 80) inside the cluster
a Service (Type NodePort) that targeting to the ubuntu pod (Port 8081), NodePort 32444
forwarding the service port in RD UI (local Port 58444)
try to access from Ubuntu Pod to this forwarded port (58444)

[adamkpickering]

There is some background that I need to explain before I can answer your question. Container-based technologies (at least, the ones provided by Rancher Desktop) require a Linux kernel to work. Rancher Desktop can run on Windows and macOS because it effectively wraps a VM that runs Linux. So when you're running docker run ..., or kubectl apply ..., you're actually talking to docker or kubernetes inside this VM. Much of the work of developing Rancher Desktop has to do with making the communication between the host machine, which you're interacting with, and the Linux VM, seamless.

The purpose of the UI port forwarding feature is to let the user access ports that are open in the Linux VM from the host machine. This only goes one way, i.e. it's not possible to use the UI port forwarding feature to expose ports that are open on the host machine to the Linux VM, which is what you are trying to do.

I was going to say that this isn't something that k8s supports, but that would have been wrong. In the PodSpec object there is the hostNetwork key, which lets you configure a Pod to use the root network namespace of the Node it is running on. This would let you do what you want, i.e. talk to a service listening in the root network namespace from within a Pod running in k8s.

However, going back to how Rancher Desktop runs everything in a Linux VM... for this to work in Rancher Desktop we would have had to introduce something that would pass traffic from the Linux VM to things listening on the host network. I'm not sure if this should work. @Nino-K do you know?

[adamkpickering]

I see, that makes more sense. At least, up until this part:

(where i can reach it from a pod inside the cluster)

Now it sounds like you're trying to forward a port from the Linux VM so that it is accessible on the Windows host, but then you're trying to access that service from inside a pod that is running inside k3s, which is inside the Linux VM. Which doesn't make any sense. Do you have an external (i.e. not a part of Rancher Desktop) cluster or something?

You're right that ports forwarded through the UI listen on 127.0.0.1, not 0.0.0.0. This is not a bug; this is how the code is written at the moment. If you want, you could create an issue to try to get this changed. I'm not sure what the outcome of that would be though - I'm not sure that this is something we want to do.

[TheGreatBey0nd]

is inside the Linux VM. Which doesn't make any sense

Imagine you have multiple pods running in the cluster. All of them should be able to reach each other. If we then wanted to debug a service locally with Eclipse, we would have to change the URLs of all services. However, if we were to provide all services with port forwarding, then I could simply replace a service with the local version (eclipse). And this is exactly how we currently use Docker Desktop

[adamkpickering]

I think I understand now. So Docker Desktop allows Pods to access things that are listening on the host's network? Honestly that sounds bizarre to me - it breaks the whole containerization model. But if it does, it does... @jandubois have you heard of this? Docker Desktop does this, so Rancher Desktop probably should too.

Some suggestions: you could look at using the hostNetwork key. You could also experiment with LoadBalancer or NodePort services. No guarantees on either of those working for you though.

One final unsolicited comment: your debugging setup sounds complicated and error prone. If I were you, I'd try getting a shell inside the pod you're trying to debug, and debugging from there. That way you eliminate the possibility of port forwarding screwing up and confusing you. I know this might not be possible or comfortable with your dev environment, but I thought I'd suggest it. The likelihood of running into issues increases exponentially when you are working outside the normal use pattern on complex systems like these.

[Nino-K]

@TheGreatBey0nd we have an open issue that will address the listening IP address for nodePort and Loadbalancers.
Also, please take a look at this response which explains why it's not working in the context of docker.

@adamkpickering

Honestly that sounds bizarre to me - it breaks the whole containerization model

it is not unusual for containers or pods to access external services or even services on the host, an example of that is when a pod is trying to access a DB hosted locally on a user's machine.

@TheGreatBey0nd can you run this firewall rule to see if it rectifies the issue? if not, then I can say the issue is how we are binding to 127.0.0.1 which is being addressed in here: #3093.

[adamkpickering]

Interesting, I had no idea. How would a program running in a pod access something running on the node then? Via the node's public IP address? Via coreDNS as described here?

[TheGreatBey0nd]

can you run #3326 (comment) firewall rule to see if it rectifies the issue?

I´ve testet the rule, but the behavior didn´t change.

[TheGreatBey0nd]

my problem was solved in Version 1.8.1

[adamkpickering]

Closing this discussion as the problem appears to be resolved.