Rootless, bridge network and reaching a host service from a container

[andreaippo]

Issue Description

Impossible to get rootless + bridge network + container-to-host connectivity - using pasta

Steps to reproduce the issue

Hi,

I am struggling with the following:

I have a compose file with a bunch of services (Kafka, Mongo, etc) and running on a custom bridge network I created manually (let's call it my-bridge).

I then have a docker file that builds and runs a Java webapp (my-service-a) on my-bridge:

[...]
docker run -p 8090:8080 --name=my-service-a --replace --net=my-bridge_default --add-host=host.docker.internal:host-gateway -d org.me/my-service-a

my-service-a needs to be able to communicate with those compose services (hence on the same network), e.g. consume Kafka messages.

my-service-a also exposes a REST API (which I consume from my host's browser or postman at localhost:8090). Some operations of this API cause my-service-a to perform REST calls to another app (my-service-b), which is running on my host instead, as a plain old JVM app.

my-service-a uses host name host.docker.internal in its configuration to indicate the host of my-service-b, so that it relies on name resolution and doesn't need to know a fixed IP address.

I'm quite sure that this used to work fine until very recently, so I dunno if my distro changed something, but the bottom line is, now this doesn't work anymore.

When starting my-service-a, I get this:

Error: failed to create new hosts file: unable to replace "host-gateway" of host entry "host.docker.internal:host-gateway": host containers internal IP address is empty

So I did some searching and stumbled upon some Issues in this project (e.g. #22653). I tried to apply the first recommendation mentioned here, to no avail.

I created ~/.config/containers/containers.conf with the following (how can I be sure this change is effective?):

[network]
default_rootless_network_cmd = "pasta"
pasta_options = ["--map-gw"]

Next, I have started my-service-a again, this time without the extra host (is this ok?):

docker run -p 8090:8080 --name=my-service-a --replace --net=my-bridge_default -d org.me/my-service-a

And I have changed my-service-a's app configuration so that instead of looking for my-service-b at host.docker.internal, it uses host.containers.internal (here also, is this ok?)

Now my-service-a starts just fine, but is unable to resolve host.containers.internal, therefore it's not forwarding calls to my-service-b:

Caused by: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'host.containers.internal' [A(1)] and search domain query for configured domains failed as well: [dns.podman]

What am I doing wrong?

Thanks

P.S. I have noticed that netavark-dhcp-proxy.socket/service were not running, so I enabled those via systemd, so that they start on every boot. Not sure it's correct/needed.

Describe the results you received

Error: failed to create new hosts file: unable to replace "host-gateway" of host entry "host.docker.internal:host-gateway": host containers internal IP address is empty at container startup

Describe the results you expected

Container starts on the assigned bridge network and is also able to reach a service running on the host

podman info output

host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-1.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 94.41
    systemPercent: 1.31
    userPercent: 4.28
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: opensuse-tumbleweed
    version: "20240823"
  eventLogger: journald
  freeLocks: 1206
  hostname: andromeda
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.10.5-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 1043873792
  memTotal: 32920047616
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.1.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.1-1.1.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: crun-1.15-1.1.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-20240814.61c0b0d-1.1.x86_64
    version: |
      pasta 20240814.61c0b0d-1.1
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 69784563712
  swapTotal: 69789020160
  uptime: 1h 24m 44.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /home/andrea/.config/containers/storage.conf
  containerStore:
    number: 20
    paused: 0
    running: 7
    stopped: 13
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrea/.local/share/containers/storage
  graphRootAllocated: 707194257408
  graphRootUsed: 322045759488
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 437
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/andrea/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1724314016
  BuiltTime: Thu Aug 22 10:06:56 2024
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

opensuse Tumbleweed with SELinux enforcing

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

Just report these spam comments to github, when I see them I remove the comments but it seems github also quickly removes them (doesn't seem to prevent them from being posting in the first place unfortunately)

[Luap99]

Lets keep the discussion in one place please
#23776