Host unreachable from container with bridge network on Podman v5 #22653
Comments
n-hass
changed the title
|
Can you provide a |
|
@mheon sure. See below |
|
I am experiencing the same issue. I am not able to connect to any port that is in use on the host. I am able to ping the host. I am also able to connect to a port that is exposed by another container. This is inside the container. |
|
Can you try the 5.0 install with a container created with the |
|
@mheon Yep. |
|
@Luap99 Are we aware of this one on the Pasta side, or is this new? |
|
I was also using pasta before on 4.x. I upgraded from fredora 39 to 40 and podman 4.x to 5.x. pasta is set in the ~/.config/containers/containers.conf. ~/.config/containers/containers.conf Without the pasta_options set it also fails to connect. |
|
i am experiencing the same issue, how to fix it? |
|
Experiencing the same issue here. My container host updated from CoreOS 39 to 40 yesterday. I run a certificate authority container with host network mode, and a Traefik container that is connected to a bridge network and an internal network that has all of the backend services connected for proxying through Traefik. After the update, Traefik gets connection refused when trying to connect to the CA container on the host network. |
|
I am not sure if this is the exact same issue. Whether I use bridge or host networking, some containers are not accessible using the host IP but they are using the container IP. For example, running nginx on 443 will result in connection refused using host IP and succeeds using the container IP regardless if I'm using host network or bridge with port mapping. Now the interesting part, I reboot the host and the problem switches to a different set of containers that become inaccessible using the host IP while the nginx starts working. Each reboot seems to transfer the problem to a different container, I haven't figured out any pattern. Could be a different issue though as I may have had the issue on podman 4.X before upgrading. No system firewall is enabled. Edit: I should add containers are being run from root user / systemd. |
Yes I know this, these are two issues really. Second, bridge as rootless adds the wrong host.containers.internal ip. I fixed it for |
Maybe the work in progress to make forwarding more flexible will make this less of a problem, as we'll probably be able to say stuff like "splice container socket to host socket connected to port 5000 and source 2001:fd8::1`. But anyway, it's not necessarily going to be magic and intuitive, so, let's consider the current situation, as it won't be necessarily very different with this regard. There are pretty much five ways to connect to a service running on the host, with pasta:
I think it would help if we pick one of those as recommended solution, eventually. |
|
I am going with |
Luap99
added
network
That is what the code is supposed to do today when trying to create a ip for host.containers.internal, however for the bridge network mode it does not work currently. I try to fix this part. However I think the actual issue is still #19213, we need a way to map a arbitrary ip in the netns to the host ip and than expose it as host.containers.internal to the container. |
...that is, just like |
Yes it is important that the address is not localhost, it must be impossible for such a mapping to reach the hosts localhost address, it must only work for services listing on the external interface. |
|
I faced an issue with I have a pod that runs a container with a This
|
|
@dimazest I don't see how this is related to this issue here? |
We have to exclude the ips in the rootless netns as they are not the host. Now that fix only works if there are more than one ip one the host available, if there is only one we do not set the entry at all which I consider better as failing to resolve this name is a much better error for users than connecting to a wrong ip. It also matches what --network pasta already does. The test is bit more compilcated as I would like, however it must deal with both cases one ip, more than one so there is no way around it I think. Fixes containers#22653 Signed-off-by: Paul Holzinger <[email protected]>
We have to exclude the ips in the rootless netns as they are not the host. Now that fix only works if there are more than one ip one the host available, if there is only one we do not set the entry at all which I consider better as failing to resolve this name is a much better error for users than connecting to a wrong ip. It also matches what --network pasta already does. The test is bit more compilcated as I would like, however it must deal with both cases one ip, more than one so there is no way around it I think. Fixes containers#22653 Signed-off-by: Paul Holzinger <[email protected]>
|
@lazyzyf Maybe this helps: I am experiencing this issue with service running with I start a HTTP server on the host with I execute I execute
I set the following in [network]
default_rootless_network_cmd = "slirp4netns"See https://blog.podman.io/2024/03/podman-5-0-breaking-changes-in-detail/ section "Pasta default for rootless networking". After Instead of reverting to [network]
pasta_options = ["--address", "10.0.2.0", "--netmask", "24", "--gateway", "10.0.2.2", "--dns-forward", "10.0.2.3"]This appears to work independently of the IP address and network mask used by the container. My system: I came here from #22724. |
|
Is this actually fixed? Although @sbrivio-rh 's answer seems like there are so many solutions but I haven't been able to make any of them work. It seems like all the solutions in this issue involve not using a bridge network, even though the title specifically calls for it. to I want to do something that seems so easy: proxy a webapp from the host, using a caddy container which is in a bridge network with other containers. I can't be the only one strugling with this. |
Well, it looks like the implications of pasta copying host addresses into the containers are catching some users by surprise, so we have to come up with something nicer, for example giving the ability to configure an arbitrary address representing the host (see #22653 (comment)). Preconditions for that are work in progress.
What happens if you assign a secondary address on the host, and use that to reach the host from your container? That should work even with a bridge and pasta or slirp4netns in between. |
|
Thanks, indeed that works! |
This solve my issue as well |
@sbrivio-rh How do I actually configure that (podman version 5.2.1)? In my case this is the best solution because I actually want the traffic to look local and I know the ports (ports 80 and 443), but adding a fails with From the PR merging passt support I vaguely remember that Podman passes some default options to passt, including |
|
This needs to be |
|
Didn't even think about that. Now I'm asking myself why I didn't, it's kinda obvious 🙈 Sorry for the noise! However, unfortunately it's still not working. My webserver is running in another container (root and with "daniel-rudolf.de" resolves to 116.203.33.181, and 116.203.33.181 is also copied to the container just fine (checked using What am I missing? |
I think what doesn't work in your case is that you're binding ports 80 and 443 using a specific address, 116.203.33.181, but with pasta's You can either bind those ports with an additional By default, the host is represented on the tap device (not on |
Thanks @sbrivio-rh! Unfortunately it still doesn't work. The webserver container is now additionally listening on Even if I change the webserver container to I'm obviously doing something wrong, any hints? 😕
I'm not 100% sure whether I understand you right here. What you're saying is that by using I tested it with However, this poses a practical issue: I then need a DNS server (or a
You mean by adding another network interface and using this interface's address? Because adding another address to the same interface shares the same faith as the public IP address after restarting the container: it is copied to the container. I just tested it by using the IP address of my wireguard interface and it indeed works (also requiring |
In the upcoming podman v5.3 release we will set --map-guest-addr by default for pasta and add the host.containers.internal host entry to that address (#23791) |
That's a good thing for sure, but having some hostname for If there's really no other solution I'll unavoidably go with it (switching back to |
Wait, you shouldn't use that address as destination, from the container. You should either:
Correct.
Right, in your case, given that you want connections to look like local ones, there's no real advantage.
That's a new use case, I never thought of it. Well, I run virtual machines (with passt(1)) on passt.top and I often need to fetch git://passt.top/passt/ from those virtual machines, so I'm just used to add an entry to But for containers, this is new to me, and I guess we could actually bind ports in the container without binding specifically to By the way, for the moment being, you don't really need a DNS server just for a few domains, you could add entries to /etc/hosts that are specific for your containers. |
Ah! That's it! 🙈 I just tried the following solutions (as reference for others) and they both work flawlessly, thanks @sbrivio-rh! 👍 ❤️
However, using the gateway address (i.e. without I'm not going to use that variant (because
That would be just perfect 👍 In the meantime I'll go with
@Luap99 Since I finally understand what's going on I'd like to suggest adding a Podman CLI option similar to |
That should already be possible via |
Ah, good to know. Didn't know that this exists because it's not documented yet. Just opened a PR for that, see #24043. Checked the code and I assume that it will use |
On this topic, @Luap99, we were wondering: right now, with I know that Podman doesn't use it, but you raised this issue with other options in the past: would you consider a security issue the fact that outbound traffic (container to host) to a non-loopback address would be mapped, on the host, to a loopback address as source? |
You did not show the definition of network you have in your compose file. Very often people have something like this How does |
|
Thanks! The new defaults in Podman 5.3 fixed this for me -- I no longer need any overrides in |
Issue Description
I am running a web service on my host, which I would expect could be accessed from a bridge networked container.
This works on Podman v4.7.2:
podman run --rm --network=bridge docker.io/mwendler/wget host.containers.internal:8091The same does not work on v5.0.2, with
Connecting to 10.1.26.100:8091... failed: Connection refused.Here, 10.1.26.100 is the host's eth0 address (
host.containers.internal), but the result is the same if i use the bridge's gateway IP.Steps to reproduce the issue
Steps to reproduce the issue
podman runwith--network=bridgehost.containers.internalor the bridge interface's gateway IPConnection refusederrorDescribe the results you received
Connections to host from a container in bridge network mode are refused under Podman v5.0.2 when previously on v4 this was not the case.
Describe the results you expected
Container in bridge network mode can connect to the host using
host.containers.internalpodman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
Environment is a NixOS host.
Additional information
No response
The text was updated successfully, but these errors were encountered: