Wrong permissions: Rootless Quadlet in combination with Volumes

[j1nx]

Issue Description

I am using Podman to rootless run our framework containerized. For this we have the below quadlet .container file;

ovos@OpenVoiceOS:~ $ cat .config/containers/systemd/ovos-phal.container
[Unit]
Description=OpenVoiceOS PHAL Service
After=local-fs.target
After=network-online.target
Wants=network-online.target
Wants=pipewire.service
After=pipewire.service
Wants=ovos-bus-server.service
After=ovos-bus-server.service

[Container]
ContainerName=ovos-phal
Image=docker.io/smartgic/ovos-phal:alpha
#Pull=never
AutoUpdate=registry
LogDriver=journald
HostName=ovos-phal
Network=host
Timezone=local
UserNS=keep-id:uid=%U
EnvironmentFile=%h/.config/containers/env
AddDevice=-/dev/snd
Volume=%h/ovos/config:/home/ovos/.config/mycroft:z
Volume=ovos_local_state:/home/ovos/.local/state/mycroft:z
Volume=%h/ovos/tmp:/tmp:z
Volume=%t/pipewire-0:/run/user/1000/pipewire-0:ro
Volume=ovos_venv_phal:/home/ovos/.venv

[Service]
TimeoutStartSec=900
Restart=always

ovos@OpenVoiceOS:~ $ cat .config/containers/systemd/ovos-phal.container.d/AddDevice.conf
[Container]
AddDevice=-/dev/gpiomem
AddDevice=-/dev/ttyAMA0
AddDevice=-/dev/spidev0.0
AddDevice=-/dev/spidev0.1

ovos@OpenVoiceOS:~ $ cat .config/containers/systemd/ovos-phal.container.d/Mycroft.conf
[Container]
Volume=/etc/mycroft/mycroft.conf:/etc/mycroft/mycroft.conf:ro,z

ovos@OpenVoiceOS:~ $ cat .config/containers/systemd/ovos_venv_phal.volume
[Volume]
User=1000
Group=1000
Copy=true
Label=name=ovos_venv_phal

If I start the container with the above configuration, the Volumes are created with the first available UID from /etc/subuid and /etc/subgid

ovos@OpenVoiceOS:~ $ ls -la /home/ovos/.local/share/containers/storage/volumes/
total 16
drwx------ 4 ovos   ovos   4096 Aug 24 10:47 .
drwxr-xr-x 9 ovos   ovos   4096 Aug 24 10:47 ..
drwx------ 3 100000 100000 4096 Aug 24 10:47 ovos_local_state
drwx------ 3 100000 100000 4096 Aug 24 10:47 ovos_venv_phal

ovos@OpenVoiceOS:~ $ sudo cat /etc/subuid
ovos:100000:65536

ovos@OpenVoiceOS:~ $ sudo cat /etc/subgid
ovos:100000:65536

ovos@OpenVoiceOS:~ $ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

ovos@OpenVoiceOS:~ $ id
uid=1000(ovos) gid=1000(ovos) groups=1000(ovos),4(adm),5(tty),7(lp),10(wheel),18(dialout),28(video),29(audio),103(pipewire),107(systemd-journal),110(systemd-timesync),111(input),1001(i2c),1002(spi),1003(gpio),1004(network)

However I can not get in because of that and the _data folder inside those volumes are using starting subuid + UID -1

ovos@OpenVoiceOS:~ $ ls -la /home/ovos/.local/share/containers/storage/volumes/ovos_venv_phal/
ls: cannot open directory '/home/ovos/.local/share/containers/storage/volumes/ovos_venv_phal/': Permission denied
ovos@OpenVoiceOS:~ $ sudo ls -la /home/ovos/.local/share/containers/storage/volumes/ovos_venv_phal/
total 12
drwx------ 3 100000 100000 4096 Aug 24 10:47 .
drwx------ 4 ovos   ovos   4096 Aug 24 10:47 ..
drwxr-xr-x 5 100999 100999 4096 Aug 24 02:19 _data

Also confirmed by executing into the container

ovos@OpenVoiceOS:~ $ podman exec --tty --interactive ovos-phal bash
ovos@ovos-phal:~$ pwd
/home/ovos
ovos@ovos-phal:~$ ls -la
total 40
drwxr-xr-x 6 ovos ovos            4096 Aug 24 10:58 .
drwxr-xr-x 3 root root            4096 Aug 24 02:19 ..
-rw-r--r-- 1 ovos ovos              44 Aug 24 10:58 .asoundrc
-rw-r--r-- 1 ovos ovos             220 Mar 29 21:21 .bash_logout
-rw-r--r-- 1 ovos ovos            3526 Mar 29 21:21 .bashrc
drwxr-xr-x 4 ovos ovos            4096 Aug 24 10:58 .cache
drwxr-xr-x 5 ovos ovos            4096 Aug 24 10:58 .config
drwxr-xr-x 4 ovos ovos            4096 Aug 24 02:21 .local
-rw-r--r-- 1 ovos ovos             807 Mar 29 21:21 .profile
drwxr-xr-x 5  999 systemd-journal 4096 Aug 24 02:19 .venv

As a remark the host and the container/image use the same username "ovos" and userid "1000"

Searching online does show similar issues with quadlet and volumes always quickly followed by a PR to fix it, so I am not sure this is a new bug, an old bug that has resurfaced or that I am just missing something or misconfigured something on either the podman configuration, the host or both. Hence the reason I started this issue.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create / Download the .container setup for your user as described above
2. Manually pull the image from docker.io (it is freely available)
3. Start the user service

Describe the results you received

The volume itself and the _data folder inside are created with uid 100000 and 101000

Describe the results you expected

I expect the volume including the _data folder inside to be created with subuid starting number + UID ( 101000 in my case )

podman info output

ovos@OpenVoiceOS:~ $ podman info
host:
  arch: arm64
  buildahVersion: 1.37.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 96.68
    systemPercent: 0.88
    userPercent: 2.45
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: ovos
    variant: ovos-rpi4-64-buildroot
    version: 0.1.1
  eventLogger: journald
  freeLocks: 2045
  hostname: OpenVoiceOS
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.6.45-ovos
  linkmode: dynamic
  logDriver: journald
  memFree: 133824512
  memTotal: 1744224256
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: Unknown
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 1.16.1
      commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 575066112
  swapTotal: 575590400
  uptime: 0h 27m 58.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /home/ovos/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.11
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
  graphRoot: /home/ovos/.local/share/containers/storage
  graphRootAllocated: 490808864768
  graphRootUsed: 1552146432
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /home/ovos/.local/share/containers/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/ovos/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: 5.2.2
  GoVersion: go1.21.13
  Os: linux
  OsArch: linux/arm64
  Version: 5.2.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Custom build minimal-OS utilizing Buildroot.
https://github.com/OpenVoiceOS/ovos-buildroot

Additional information

No response

[rhatdan]

The volume is created via root of the user namespace, which you changed to be the first UID of the /etc/subuid with the line

Userns: keep-id:u=%U

Which tells Podman to not use the default usernamespace but an alternative one where your UID (1000) is mapped to %U in the quadlet.

podman unshare cat /proc/sys/uid_map

Is showing the default user namesapace, but your container was remapped to the keep-id one.

In order to manipulate the volume, from your homedir, you need to enter the default user namespace with podman unshare

Do you want the volumes to be created with the default UID of the container? IE Your UID (1000)? and not root of the container, you could try to add U to the volume lines.

      The :U suffix tells Podman to use the correct host UID and GID based on
       the UID and GID within the container, to change recursively  the  owner
       and  group  of  the source volume. Chowning walks the file system under
       the volume and changes the UID/GID on each  file.  If  the  volume  has
       thousands of inodes, this process takes a long time, delaying the start
       of the container.

Not sure if this will work unless the container is defaulting to run as the : %U user.

[j1nx]

Thank you very much for the additional explaination @rhatdan I have misunderstand that part of the userns documentation.

Anyhow, that is all fine then. I do not really have to access the volumes from the host, so will leave it like it is.

But then onto the second issue. Why is the "_data" folder inside created with UID 999 instead of 1000 ? (and 100999 instead of 101000 on the host)

ovos@OpenVoiceOS:~ $ sudo ls -la /home/ovos/.local/share/containers/storage/volumes/ovos_venv_phal/
total 12
drwx------ 3 100000 100000 4096 Aug 24 10:47 .
drwx------ 4 ovos   ovos   4096 Aug 24 10:47 ..
drwxr-xr-x 5 100999 100999 4096 Aug 24 02:19 _data

ovos@OpenVoiceOS:~ $ podman exec --tty --interactive ovos-phal bash
ovos@ovos-phal:~$ pwd
/home/ovos
ovos@ovos-phal:~$ ls -la
total 40
drwxr-xr-x 6 ovos ovos            4096 Aug 24 10:58 .
drwxr-xr-x 3 root root            4096 Aug 24 02:19 ..
-rw-r--r-- 1 ovos ovos              44 Aug 24 10:58 .asoundrc
-rw-r--r-- 1 ovos ovos             220 Mar 29 21:21 .bash_logout
-rw-r--r-- 1 ovos ovos            3526 Mar 29 21:21 .bashrc
drwxr-xr-x 4 ovos ovos            4096 Aug 24 10:58 .cache
drwxr-xr-x 5 ovos ovos            4096 Aug 24 10:58 .config
drwxr-xr-x 4 ovos ovos            4096 Aug 24 02:21 .local
-rw-r--r-- 1 ovos ovos             807 Mar 29 21:21 .profile
drwxr-xr-x 5  999 systemd-journal 4096 Aug 24 02:19 .venv

The ovos user with UID 1000 in the container does not have permission to write to the volume?

[j1nx]

Thank you.

Using;

UserNS=keep-id:uid=%U,gid=%G

together with;

Volume=ovos_venv_phal:/home/ovos/.venv:U

Works as expected.

The volumes are created with UID 100000 and within the volume the _data folder and everything recursive underneath that with UID 101000 which resolves to the user ovos both on the host and the container.