[rootless][quadlet] Getting OCI permission denied errors on multiple platforms

[zenlord]

Hi,
Moving from using a set of bash scripts to ansible + quadlet, and at the same time moving from Debian bookworm as the host OS to something that carries a more recent version of podman and netavark (tried both Archlinux v5.3 and Debian Trixie v5.2.5)

I spent 2.5 days on the Archlinux host (various files systems, all storage drivers, etc), and this morning I ran a desperate attempt by just installing Debian Trixie and re-running the ansible playbook to only find out that the errors are the same:

nov 13 09:26:11 nucserver systemd[12248]: Started libpod-e5b21316da413ca1d7b4748b678804f402dea3ce542b2260153d1ac4666727b4.scope - libcrun container.
nov 13 09:26:11 nucserver systemd[12248]: Stopped libpod-e5b21316da413ca1d7b4748b678804f402dea3ce542b2260153d1ac4666727b4.scope - libcrun container.
nov 13 09:26:11 nucserver conmon[13403]: conmon e5b21316da413ca1d7b4 <nwarn>: runtime stderr: open `/srv/containers/hassio/.local/share/containers/storage/overlay/ec3842f4cb73503a35060130322cc4c73537c091058979e9666f67eb635d8244/merged`: Permission denied
nov 13 09:26:11 nucserver conmon[13403]: conmon e5b21316da413ca1d7b4 <error>: Failed to create container: exit status 1
nov 13 09:26:11 nucserver hassio-pod[13392]: Error: starting container e5b21316da413ca1d7b4748b678804f402dea3ce542b2260153d1ac4666727b4: crun: open `/srv/containers/hassio/.local/share/containers/storage/overlay/ec3842f4cb73503a35060130322cc4c73537c091058979e9666f67eb635d8244/merged`: Permission denied: OCI permission denied
nov 13 09:26:11 nucserver podman[13392]: 2024-11-13 09:26:11.880363979 +0100 CET m=+0.274081236 pod stop 1c40a9a1002c5b70cbbd64867599ede1ea41ed45929343b998637af07a66fcfb (image=, name=hassio)

Which leaves me two potential options: either this image is somehow refusing to play ball, or the quadlets I have generated with ansible are not what they are supposed to be. Some of my attempts involving manipulating the container quadlet to run independently from the pod, but the result was the same.

# /run/user/5001/systemd/generator/hassio-pod.service
# Automatically generated by /usr/lib/systemd/user-generators/podman-user-generator
# 
[X-Pod]
PodName=hassio
Network=pasta:--ipv4-only,-a,10.0.2.0,-n,24,-g,10.0.2.2,--dns-forward,192.168.2.1,-m,1500,--no-ndp,--no-dhcpv6,--no-dhcp
PublishPort=8123:8123/tcp
PublishPort=5353:5353/tcp
PublishPort=5353:5353/udp
PublishPort=1900:1900/udp
PublishPort=1400:1400/tcp
PublishPort=51827:51827/tcp
PublishPort=5683:5683/udp
PodmanArgs=--dns 192.168.2.1
PodmanArgs=--userns keep-id:uid=1000,gid=1000

[Install]
WantedBy=default.target

[Unit]
SourcePath=/srv/containers/hassio/.config/containers/systemd/hassio.pod
RequiresMountsFor=%t/containers
Wants=hassio-app.service
Before=hassio-app.service

[Service]
SyslogIdentifier=%N
ExecStart=/usr/bin/podman pod start --pod-id-file=%t/%N.pod-id
ExecStop=/usr/bin/podman pod stop --pod-id-file=%t/%N.pod-id --ignore --time=10
ExecStopPost=/usr/bin/podman pod rm --pod-id-file=%t/%N.pod-id --ignore --force
ExecStartPre=/usr/bin/podman pod create --infra-conmon-pidfile=%t/%N.pid --pod-id-file=%t/%N.pod-id --exit-policy=stop --replace --publish 8123:8123/tcp --publish 5353:5353/tcp --publish 5353:5353/udp --publish 1900:1900/udp --publish 1400:1400/tcp --publish 51827:51827/tcp --publish 5683:5683/udp --network=pasta:--ipv4-only,-a,10.0.2.0,-n,24,-g,10.0.2.2,--dns-forward,192.168.2.1,-m,1500,--no-ndp,--no-dhcpv6,--no-dhcp --name=hassio --dns 192.168.2.1 --userns keep-id:uid=1000,gid=1000
Environment=PODMAN_SYSTEMD_UNIT=%n
Type=forking
Restart=on-failure
PIDFile=%t/%N.pid

And

# /run/user/5001/systemd/generator/hassio-app.service
# Automatically generated by /usr/lib/systemd/user-generators/podman-user-generator
# 
[X-Container]
ContainerName=hassio-app
Environment=DOCKER_HOST=unix:///var/run/docker.sock
Environment=DISABLE_JEMALLOC=true
Image=ghcr.io/home-assistant/home-assistant
Pod=hassio.pod
Timezone=Europe/Brussels
Volume=/run/user/5001/podman/podman.sock:/var/run/docker.sock
Volume=/srv/containers/hassio/hassio-config:/config
PodmanArgs=--security-opt apparmor=unconfined
AutoUpdate=registry

[Install]
WantedBy=default.target

[Unit]
Wants=network-online.target
After=network-online.target
SourcePath=/srv/containers/hassio/.config/containers/systemd/hassio-app.container
RequiresMountsFor=%t/containers
RequiresMountsFor=/run/user/5001/podman/podman.sock
RequiresMountsFor=/srv/containers/hassio/hassio-config
BindsTo=hassio-pod.service
After=hassio-pod.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -v -f -i --cidfile=%t/%N.cid
ExecStopPost=-/usr/bin/podman rm -v -f -i --cidfile=%t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=hassio-app --cidfile=%t/%N.cid --replace --rm --cgroups=split --tz=Europe/Brussels --sdnotify=conmon -d -v /run/user/5001/podman/podman.sock:/var/run/docker.sock -v /srv/containers/hassio/hassio-config:/config --label io.containers.autoupdate=registry --env DISABLE_JEMALLOC=true --env DOCKER_HOST=unix:///var/run/docker.sock --pod-id-file %t/hassio-pod.pod-id --security-opt apparmor=unconfined ghcr.io/home-assistant/home-assistant

Any help would be very much appreciated

[zenlord]

I can confirm I have the same error with starting a hello-world container based on the above hassio-app.service (having only removed the reference t the pod and the image name).

Also changing the the log_driver back to k8s-file instead of journald and removing the log_tag did not change anything. The containers.conf and storage.conf files are close to vanilla, and even after deleting the config files in the $HOME directory and running a podman system reset, the errors remain the same.

$ podman info
host:
  arch: amd64
  buildahVersion: 1.37.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.12-3_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 99.47
    systemPercent: 0.18
    userPercent: 0.34
  cpus: 4
  databaseBackend: sqlite
  distribution:
    codename: trixie
    distribution: debian
    version: unknown
  eventLogger: journald
  freeLocks: 2048
  hostname: nucserver.aplusv.be
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 5001
      size: 1
    - container_id: 1
      host_id: 500100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 5001
      size: 1
    - container_id: 1
      host_id: 500100000
      size: 65536
  kernel: 6.11.5-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 6968913920
  memTotal: 8153088000
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.12.2-1_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark_1.12.1-3_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: crun_1.18.2-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.18.2
      commit: 00ab38af875ddd0d1a8226addda52e1de18339b5
      rundir: /run/user/5001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20241030.ee7d0b6-1_amd64
    version: |
      pasta 0.0~git20241030.ee7d0b6-1
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/5001/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1+b1_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 6196031488
  swapTotal: 6196031488
  uptime: 2h 51m 35.00s (Approximately 0.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /srv/containers/hassio/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /srv/containers/hassio/.local/share/containers/storage
  graphRootAllocated: 110441844736
  graphRootUsed: 1660555264
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/5001/containers
  transientStore: false
  volumePath: /srv/containers/hassio/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.5
  Built: 1729890366
  BuiltTime: Fri Oct 25 23:06:06 2024
  GitCommit: ""
  GoVersion: go1.23.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.5

I am entering the user session using machinectl shell hassio@ and have tried unsetting any env vars I had set...

[zenlord]

Switching from crun to runc has helped me run the hello-world container and the uptime-kuma container. The Home Assistant container is also failing with runc (in rootless mode, privileged and unprivileged) - but this is very likely not an error in any package.

Happy that I'm finally on my way to a solution - Don't know what to think of runc vs crun - thought crun was recommended so never questioned that choice before...