Published ports can't be reached when attaching to second network and running as systemd service

[theCalcaholic]

Issue Description

So, this is a strange one.. I'm trying to run a webserver (Caddy) as systemd service that is connected to two podman networks, one of which has been created externally (to the service).

However, I noticed that the published container ports (80 and 443) are unavailable when run as systemd unit, but the exact same command works without issues when run directly from a root tty (in both cases, the container is run as root).

The problem goes away if I do not connect it to the previously created network

Steps to reproduce the issue

Steps to reproduce the issue

1. Create and start a service like this:

   cat <<EOF > /etc/systemd/system/caddy.service
   [Service]
   Type=exec
   NotifyAccess=all
   ExecStartPre=/bin/bash -c '/usr/bin/podman network inspect public >/dev/null 2>&1 || /usr/bin/podman network create public'
   ExecStartPre=-/usr/bin/podman rm caddy
   ExecStartPre=/usr/bin/podman create \
       --network public \
       -p 80:80 \
       --name caddy \
       docker.io/caddy:2
   # Network 'internal' has been created previously by another service
   ExecStartPre=/usr/bin/podman network connect internal caddy
   ExecStart=/usr/bin/podman start --attach caddy
   ExecStop=/usr/bin/podman rm nca-caddy
   EOF
   systemctl daemon-reload
   systemctl start caddy

2. Test whether you can reach the container at its published port

   curl -v http://localhost:80

   -> observe a timeout

3. Test whether you can reach the container at its ip address

   curl -v "http://$(podman inspect caddy -f '{{.NetworkSettings.Networks.internal.IPAddress}}')" # <-- observe a timeout
   curl -v "http://$(podman inspect caddy -f '{{.NetworkSettings.Networks.public.IPAddress}}')"   # <-- successfully reaches the container

4. Disconnect the container from the "internal" network

   podman network disconnect internal caddy

5. Test whether you can reach the container at its published port

   curl -v http://localhost:80

   --> successfully retrieve the default caddy page

6. Stop the service and replicate the container without systemd

   systemctl stop caddy
   podman rm caddy #just to be sure
   podman create \
       --network public \
       -p 80:80 \
       --name caddy \
       docker.io/caddy:2
   podman network connect internal caddy
   podman start caddy

7. Test whether you can reach the container at its published port

   curl -v http://localhost:80

   --> successfully retrieve the default caddy page

8. Test whether you can reach the container at its ip address

   curl -v "http://$(podman inspect caddy -f '{{.NetworkSettings.Networks.internal.IPAddress}}')" # <-- observe a timeout
   curl -v "http://$(podman inspect caddy -f '{{.NetworkSettings.Networks.public.IPAddress}}')"   # <-- successfully reaches the container

Describe the results you received

The port binding doesn't work and the container can't be reach at its published port.

Describe the results you expected

The port binding should just work and the container should be accessible at http://localhost

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 95.47
    systemPercent: 1.29
    userPercent: 3.24
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: skiff
    version: 7e23644a
  eventLogger: journald
  hostname: skiffos-7e23644a
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.10.11-200.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 32100884480
  memTotal: 32333565952
  networkBackend: cni
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8580943872
  swapTotal: 8589930496
  uptime: 17h 21m 13.00s (Approximately 0.71 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /mnt/persist/skiff/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 277700661248
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /mnt/persist/skiff/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: 4.5.1
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

Yes

Privileged Or Rootless

None

Upstream Latest Release

No

Additional environment details

podman is running inside an incus container. The rootfs has been built with buildroot (hence I'm unfortunately stuck with the podman version that's supported by buildroot).

Additional information

No response