"bind: address already in use" error restarting the container

[mikea]

Issue Description

podman restart fails with "bind: address already in use" error.

Steps to reproduce the issue

podman restart dashy fails with "bind: address already in use".

podman restart debug log: https://gist.github.com/mikea/141b43ea49a0990c7de84ae8581bc15b

Recreating container from scratch works.

podman info output

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 32
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: poweredge
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.0-52-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 57551052800
  memTotal: 101281112064
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 68651315200
  swapTotal: 68651315200
  uptime: 14h 27m 40.74s (Approximately 0.58 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 16
    paused: 0
    running: 11
    stopped: 5
  graphDriverName: zfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Compression: lz4
    Parent Dataset: rpool/ROOT/ubuntu_kqxyo6/var/lib
    Parent Quota: "no"
    Space Available: "107717689344"
    Space Used By Parent: "20058980352"
    Zpool: rpool
    Zpool Health: ONLINE
  imageStore:
    number: 13
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.18.1
  OsArch: linux/amd64
  Version: 3.4.4

Podman in a container

No

Privileged Or Rootless

None

Additional information

My systemd unit file:

[Unit]
After=network-online.target
Description=dashy
RequiresMountsFor=/run/containers/storage
Wants=network-online.target

[Install]
WantedBy=default.target

[Service]
Type=forking
ExecReload=podman --log-level=debug restart dashy
ExecStart=podman run --name=dashy --replace --conmon-pidfile=%t/%n.pid --cidfile=%t/%n.cid -d=true -p 127.0.0.1:10201:8080 -v /etc/dashy.yml:/app/user-data/conf.yml:ro -v /etc/timezone:/etc/timezone:ro -v /etc/localtime:/etc/localtime:ro docker.io/lissy93/dashy:latest
ExecStartPre=/usr/bin/rm -f %t/%n.pid %t/%n.cid
ExecStop=/usr/bin/sh -c "podman rm -f `cat %t/%n.cid`"
PIDFile=%t/%n.pid
Restart=on-failure
TimeoutStopSec=70
KillMode=control-group
Environment=PODMAN_SYSTEMD_UNIT=%n

[mikea]

I see quite a few other "address already in use" issue reporting this, but unfortunately none of them has any resolution. My apologies if I have missed anything.

[Luap99]

First of 3.4 is ancient please try this with the latest version as this is the only one we support upstream.

Second, never use podman restart if you runt he container via systemd. you should to run systemctl restart <unit>