incorrect ownership with configmap/secret content from container, when related volume mounted with restrictive defaultMode and pod running with userns auto mode

[flabatut]

Issue Description

I'm using podman kube play to create a pod and associated volume, made from a configmap (or a secret) resource.
I run this pod, as root, using --userns=auto mode.
Whenever such volume is created:

• _data folder is owned by one of the subordinate uid/gid tied with containers account.
• volume files, within _data, are owned by root, executing podman

Whenever my volume is declared with restrictive permissions for others, using defaultMode:

  volumes:
  - name: test
    configMap:
      defaultMode: 0750
      name: "test"

I'm no longer able to consume volume's content from a container standpoint, lack of permissions prevent any access/modification.

Steps to reproduce the issue

1. create resources:

cat <<EOF >test.configmap.yml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: test
data:
  readme.txt: lorem ipsum
EOF

cat <<EOF >test.pod.yml
---
apiVersion: v1
kind: Pod
metadata:
  name: "test"
spec:
  containers:
  - name: "test"
    image: docker.io/alpine:latest
    command: ["sleep", "3600"]
    volumeMounts:
    - name: "test"
      mountPath: /test
  volumes:
  - name: "test"
    configMap:
      name: "test"
      defaultMode: 0600
EOF

2. run podman kube play:

/usr/bin/podman kube play --replace --service-container=true --log-driver journald --userns auto  --configmap test.configmap.yml  test.pod.yml

3. podman volume inspect test

[
     {
          "Name": "test",
          "Driver": "local",
          "Mountpoint": "/var/lib/containers/storage/volumes/test/_data",
          "CreatedAt": "2023-12-09T00:12:05.321084238+01:00",
          "Labels": {},
          "Scope": "local",
          "Options": {},
          "UID": 100000,
          "GID": 100000,
          "MountCount": 0,
          "NeedsCopyUp": true,
          "LockNumber": 62
     }
]

4. volume permissions from host standpoint

ls -ld /var/lib/containers/storage/volumes/test/_data
drwxr-xr-x 2 100000 100000 24 Dec  9 00:12 /var/lib/containers/storage/volumes/test/_data

ls -l /var/lib/containers/storage/volumes/test/_data/
total 4
-rw------- 1 root root 11 Dec  9 00:00 readme.txt

5. volume permissions from container standpoint

podman exec -it test-test stat /test/readme.txt
  File: /test/readme.txt
  Size: 11              Blocks: 8          IO Block: 4096   regular file
Device: fe01h/65025d    Inode: 50099       Links: 1
Access: (0600/-rw-------)  Uid: (65534/  nobody)   Gid: (65534/  nobody)
Access: 2023-12-08 23:00:35.613724591 +0000
Modify: 2023-12-08 23:00:35.613724591 +0000
Change: 2023-12-08 23:00:35.613724591 +0000

podman exec -it test-test cat  /test/readme.txt
cat: can't open '/test/readme.txt': Permission denied

podman exec -it test-test chmod 666 /test/readme.txt
chmod: /test/readme.txt: Operation not permitted

podman exec -it test-test chown root:root /test/readme.txt
chown: /test/readme.txt: Operation not permitted

Describe the results you received

until some action to fix permission, at host level is performed, volume file content at container level is unusable

Describe the results you expected

volume files permissions owned by a container subordinate uid/gid rather than a host uid/gid. Differently speaking, make volume file content available as soon as pod running without any post action required.

podman info output

- architecture

# uname -a
Linux nas 6.2.0-1018-raspi #20-Ubuntu SMP PREEMPT Tue Nov 21 13:32:12 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

• os

PRETTY_NAME="Ubuntu 23.04"
NAME="Ubuntu"
VERSION_ID="23.04"
VERSION="23.04 (Lunar Lobster)"
VERSION_CODENAME=lunar

• podman info

host:
  arch: arm64
  buildahVersion: 1.33.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_1:2.1.8-1_arm64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d'
  cpuUtilization:
    idlePercent: 97.09
    systemPercent: 1.72
    userPercent: 1.18
  cpus: 4
  databaseBackend: boltdb
  distribution:
    codename: lunar
    distribution: ubuntu
    version: "23.04"
  eventLogger: journald
  freeLocks: 1966
  hostname: nas
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.2.0-1018-raspi
  linkmode: dynamic
  logDriver: journald
  memFree: 178814976
  memTotal: 8188760064
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_arm64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.6.0
    package: netavark_1.4.0-3_arm64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.12-1_arm64
    path: /usr/local/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0.20231119.4f1709d-1_arm64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.2-0ubuntu22.04+obs12.25_arm64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.4
  swapFree: 4292603904
  swapTotal: 4294963200
  uptime: 23h 45m 37.00s (Approximately 0.96 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 44
    paused: 0
    running: 44
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 429287014400
  graphRootUsed: 160218333184
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 23
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.8.0
  Built: 1701241582
  BuiltTime: Wed Nov 29 08:06:22 2023
  GitCommit: c4dfcf14874479e34b3f312f089fc5840e306258
  GoVersion: go1.20.3
  Os: linux
  OsArch: linux/arm64
  Version: 4.8.0

### Podman in a container

No

### Privileged Or Rootless

Privileged

### Upstream Latest Release

Yes

### Additional environment details

running on raspberry pi4, no virtualization

### Additional information

- occurs whenever configmap/secret volume has restrictive `defaultMode` configured
- not reproducible if `defaultMode` not declared  AND umask provides at least read access to `others`
- problem occurs once, when volume isn't yet created and is mitigated as soon as ownership modified manually
- problem arose especially with recent support addition of volumes `defaultMode` (https://github.com/containers/podman/issues/19313)
- not a blocker ; one quick workaround is to change volume files ownership from the host.
- problem reproducible with `--userns=auto` or `--userns=auto:uidmapping=0:100000:65536,gidmapping=0:100000:65536`

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@umohnani8 or @ygalblum PTAL

[flabatut]

For reference, problem solved, thanks to @rhatdan 's meaningful comment :)

Setting :U option to volume's mounPath let podman know how to correlate userns's idmap and volume content's permission:

    volumeMounts:
    - name: "test"
      mountPath: /test:U

Afterwards, problem is no longer reproducible, no post actions needed at host level to fixup content's ownership:

# podman exec -it test-test stat /test/readme.txt
  File: /test/readme.txt
  Size: 11              Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 18607005    Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-06-28 06:18:43.427719744 +0000
Modify: 2024-06-28 06:18:36.193222163 +0000
Change: 2024-06-28 06:18:36.661254353 +0000

# podman exec -it test-test cat  /test/readme.txt
lorem ipsum

# podman exec -it test-test chmod 666 /test/readme.txt
# echo $?
0
# podman exec -it test-test chown root:root /test/readme.txt
# echo $?
0