Permission denied in .local when using --userns=keep-id #23369

snyderth · 2024-07-23T00:03:38Z

Issue Description

Our use-case for podman is containerizing a production development environment. We want each developer to be able to use a container as if it were the bare-metal development system. The way we accomplish this is using the --userns=keep-id option on podman run, but some developers run into the problem where podman cannot access image layers (getting permission denied).

Steps to reproduce the issue

podman pull
podman run --pids-limit=0 -d -i -t --init --privileged --userns=keep-id --env HOME=$HOME --env SHELL=$SHELL --group-add=mock <image>

Describe the results you received

Error: creating container storage: creating read-write layer with ID "b4a6726d5bcc53585a3bd1a265877ce50b743a6db23a9c2914a4349f30762112": Stat /home/<user>/.local/share/containers/storage/overlay/93db864845d68b98ad858b4917496543786da7bdd29a0dd1885907401cd29b00/diff: permission denied

Describe the results you expected

A podman container up and running that can be exec'd into.

podman info output

host:
  arch: amd64
  buildahVersion: 1.36.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: fb8c4bf50dbc044a338137871b096eea8041a1fa'
  cpuUtilization:
    idlePercent: 99.13
    systemPercent: 0.57
    userPercent: 0.31
  cpus: 192
  databaseBackend: boltdb
  distribution:
    distribution: rhel
    version: "9.4"
  eventLogger: file
  freeLocks: 2040
  hostname: foo
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2222
      size: 1
    - container_id: 1
      host_id: 493216
      size: 65536
    uidmap:
    - container_id: 0
      host_id: <uid>
      size: 1
    - container_id: 1
      host_id: 493216
      size: 65536
  kernel: 5.14.0-427.13.1.el9_4.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 15792254976
  memTotal: 3178267447296
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.9.0-1.el9.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.9.0
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/<uid>/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240624.g1ee2eca-1.el9.x86_64
    version: |
      pasta 0^20240624.g1ee2eca-1.el9.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/<uid>/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 5361889280
  swapTotal: 5366083584
  uptime: 288h 19m 26.00s (Approximately 12.00 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/<user>/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 2
    stopped: 6
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/<user>/.local/share/containers/storage
  graphRootAllocated: 20644311859200
  graphRootUsed: 14263525134336
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/<uid>/containers
  transientStore: false
  volumePath: /home/<user>/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.1
  Built: 1719592558
  BuiltTime: Fri Jun 28 12:35:58 2024
  GitCommit: ""
  GoVersion: go1.22.4 (Red Hat 1.22.4-1.el9)
  Os: linux
  OsArch: linux/amd64
  Version: 5.1.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

ls -al /home/<user>/.local/share/containers/storage/overlay
drwx------. 5 576608 576608 92 Jun 26 17:16 93db864845d68b98ad858b4917496543786da7bdd29a0dd1885907401cd29b00
/etc/subuid/subgid entry:
<user>:493216:65536

Additional information

Removing --userns=keepid allows the container to run, but drops the user in as "root", which is undesirable. The ideal case allows each user to retain their user-id and username for tooling purposes.

SELinux is set to enforcing, but the behavior does not change when SELinux is switched to permissive.

The text was updated successfully, but these errors were encountered:

rhatdan · 2024-07-23T11:26:17Z

What are the permissions on $HOME?

@giuseppe this feels familiar?

giuseppe · 2024-07-23T11:37:57Z

Removing --userns=keepid allows the container to run, but drops the user in as "root", which is undesirable. The ideal case allows each user to retain their user-id and username for tooling purposes.

--userns=keep-id doesn't improve that. The user ID inside the container is mapped to the same ID on the host. Which is equivalent to running as "root" inside the user namespace since it is still mapped to UID on the host.

podman info says the backing file system is xfs, can you confirm that is used on the system where you've seen the issue?

snyderth · 2024-07-23T16:15:14Z

What are the permissions on $HOME?

ls -al /home
drwx------. 26 user group 4096 Jul 22 20:21 user

ls -al /home/user/.local
drwx------. 5 user group 57 Jan 26 20:23 .local

podman info says the backing file system is xfs, can you confirm that is used on the system where you've seen the issue?

It is xfs on an lvm volume.

--userns=keep-id doesn't improve that. The user ID inside the container is mapped to the same ID on the host. Which is equivalent to running as "root" inside the user namespace since it is still mapped to UID on the host.

Oops, what I said was wrong. We only want to retain the username and ensure rootless in the container. User ID can (and does) change.

rhatdan · 2024-07-23T18:35:24Z

If you chmod 701 /home/user /home/user/.local does it make it better?

snyderth · 2024-07-23T18:42:14Z

After running chmod 701 /home/user and chmod 701 /home/user/.local:
ls -al /home
drwx-----x. 39 user group 4096 Jul 22 20:25 user
ls -al /home/user
drwx-----x. 5 user group 57 Jan 26 20:23 .local

Result of podman run is the same:
Error: creating container storage: creating read-write layer with ID "4c4318fc08f87b16f700344060587f91131bc02f02d0a970c623171acc77bba9": Stat /home/user/.local/share/containers/storage/overlay/93db864845d68b98ad858b4917496543786da7bdd29a0dd1885907401cd29b00/diff: permission denied

snyderth · 2024-07-24T00:05:26Z

Something I should add: When we chown -R user:group /home/user/.local/share/containers/storage/overlay/93db864845d68b98ad858b4917496543786da7bdd29a0dd1885907401cd29b00/ we can temporarily launch the container, but it will revert after the container stops. This also has negative side-effects inside the container, though. For example, the sudo binary will become inaccessible because it is no longer owned by the root uid.

Edit: Also if it helps, this is reproducible on three identical systems running stock RHEL 9.4.

rhatdan · 2024-07-24T19:48:51Z

I would just do a podman system reset and start again, Also run a restorecon -R -v $HOME to make sure everything is labeled correctly if you are using SELinux.

Something is strange about your laptop. We are not seeing this anywhere else.

snyderth · 2024-07-24T20:28:15Z

Hmmm restorecon also results in permission denied:

restorecon -R -v $HOME
restorecon: Could not read /home/user/.local/share/containers/storage/overlay-containers/841beb6748a5ff39be58757fb3ce137cdfe785577f298f3599da1c86ffaa1fdb/userdata: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/86ebf2c5033e6e3653e56412e45b5cc8225d8820d9ce038c1bc4fd23777e3966: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/run/rpcbind: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/etc/polkit-1/rules.d: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/var/lib/nfs/statd: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/var/lib/libvirt/qemu: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/var/cache/libvirt/qemu: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/usr/share/polkit-1/rules.d: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/93db864845d68b98ad858b4917496543786da7bdd29a0dd1885907401cd29b00: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/etc/polkit-1/rules.d: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/var/lib/nfs/statd: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/var/lib/libvirt/qemu: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/var/cache/libvirt/qemu: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/usr/share/polkit-1/rules.d: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/run/rpcbind: Permission denied.

podman system reset also has permission denied:

ERRO[0002] Error removing container 841beb6748a5ff39be58757fb3ce137cdfe785577f298f3599da1c86ffaa1fdb: <nil>
ERRO[0002] Removing container 841beb6748a5ff39be58757fb3ce137cdfe785577f298f3599da1c86ffaa1fdb: cleaning up storage: removing container 841beb6748a5ff39be58757fb3ce137cdfe785577f298f3599da1c86ffaa1fdb artifacts "/home/user/.local/share/containers/storage/overlay-containers/841beb6748a5ff39be58757fb3ce137cdfe785577f298f3599da1c86ffaa1fdb/userdata/artifacts": open /home/user/.local/share/containers/storage/overlay-containers/841beb6748a5ff39be58757fb3ce137cdfe785577f298f3599da1c86ffaa1fdb/userdata: permission denied
ERRO[0010] 1 error occurred:
        * openfdat /home/user/.local/share/containers/storage/overlay/86ebf2c5033e6e3653e56412e45b5cc8225d8820d9ce038c1bc4fd23777e3966: permission denied

I think it is interesting that whatever checks file/user/group permissions is seemingly not considering the subuid/subgid ownership. Any insight on this?

rhatdan · 2024-07-24T20:32:15Z

podman unshare restorecon -R -v $HOME

rhatdan · 2024-07-24T20:33:36Z

I am surprised podman system reset is not entering the user namespace, Unless you have files in your homedir which are not owned by UIDs in your user namespace.

snyderth · 2024-07-24T20:36:06Z

podman unshare restorecon -R -v $HOME

podman unshare restorecon -R -v $HOME
restorecon: Could not read /home/user/.local/share/containers/storage/overlay-containers/841beb6748a5ff39be58757fb3ce137cdfe785577f298f3599da1c86ffaa1fdb/userdata: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/86ebf2c5033e6e3653e56412e45b5cc8225d8820d9ce038c1bc4fd23777e3966: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/run/rpcbind: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/etc/polkit-1/rules.d: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/var/lib/nfs/statd: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/var/lib/libvirt/qemu: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/var/cache/libvirt/qemu: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/8fc638aa127c3fbb221b44c6f23a4e7d7acd029351661c55d2ef982948d57652/diff/usr/share/polkit-1/rules.d: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/93db864845d68b98ad858b4917496543786da7bdd29a0dd1885907401cd29b00: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/etc/polkit-1/rules.d: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/var/lib/nfs/statd: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/var/lib/libvirt/qemu: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/var/cache/libvirt/qemu: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/usr/share/polkit-1/rules.d: Permission denied.
restorecon: Could not read /home/user/.local/share/containers/storage/overlay/9f977f05ccc75ebb226a5d3cbbbdd0a68f016097e83cae190bd206b3c5d1b5c9/diff/run/rpcbind: Permission denied.

snyderth · 2024-07-24T20:46:54Z

Looking around, I'll check on kernel.unprivileged_userns_clone=1 from this older issue: #3890 (comment)

rhatdan · 2024-07-24T20:52:55Z

I would do sudo chown -R UID:GID /home/user/.local/share/containers
Then do the podman system reset

snyderth · 2024-07-24T20:54:36Z

Looking around, I'll check on kernel.unprivileged_userns_clone=1 from this older issue: #3890 (comment)

Turns out, RHEL does not support this feature.

snyderth · 2024-07-24T21:00:32Z

I would do sudo chown -R UID:GID /home/user/.local/share/containers Then do the podman system reset

After these, my user seems to work well. I will tell my team to do this and verify that it works for everyone before closing this issue.

giuseppe · 2024-07-24T21:12:00Z

how have you created these files?

Was the storage copied from another system as root?

snyderth · 2024-07-24T21:14:51Z

how have you created these files?

Was the storage copied from another system as root?

No, these files were created from a podman pull directed at our local image registry. The original image was build using rootless podman on a bare-metal system and podman push'd to the registry.

giuseppe · 2024-07-24T21:15:55Z

How are additional IDs configured for the users on the system? Statically in /etc/subuid and /etc/subgid? Do the mappings change?

snyderth · 2024-07-24T21:22:46Z

Statically via /etc/subuid and /etc/subgid. In practice, we have a script that is run by a cronjob every 30 seconds that reads /etc/passwd and generates a new /set/sub?id. As long as the order in /etc/passwd is preserved, the mappings should stay constant. Although I can see how that would cause an issue. But wouldn't a call to podman system migrate mitigate this issue? Or does that only change newly allocated layers?

giuseppe · 2024-07-24T21:24:00Z

if these ids change, then the files in the local storage are stored with the wrong ownership, causing the issue you had

snyderth · 2024-07-24T21:37:51Z

Ok this makes a lot of sense. It looks people are able to recover by doing a podman system reset after chowning their directory back. We'll have to think of a new way to allocate subuid/subgid's. Thanks for all your help!

snyderth added the kind/bug Categorizes issue or PR as related to a bug. label Jul 23, 2024

snyderth closed this as completed Jul 24, 2024

stale-locking-app bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Oct 23, 2024

stale-locking-app bot locked as resolved and limited conversation to collaborators Oct 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Permission denied in .local when using --userns=keep-id #23369

Permission denied in .local when using --userns=keep-id #23369

snyderth commented Jul 23, 2024

rhatdan commented Jul 23, 2024

giuseppe commented Jul 23, 2024

snyderth commented Jul 23, 2024

rhatdan commented Jul 23, 2024

snyderth commented Jul 23, 2024

snyderth commented Jul 24, 2024 •

edited

Loading

rhatdan commented Jul 24, 2024

snyderth commented Jul 24, 2024

rhatdan commented Jul 24, 2024

rhatdan commented Jul 24, 2024

snyderth commented Jul 24, 2024

snyderth commented Jul 24, 2024

rhatdan commented Jul 24, 2024

snyderth commented Jul 24, 2024

snyderth commented Jul 24, 2024

giuseppe commented Jul 24, 2024

snyderth commented Jul 24, 2024

giuseppe commented Jul 24, 2024

snyderth commented Jul 24, 2024

giuseppe commented Jul 24, 2024

snyderth commented Jul 24, 2024

Permission denied in .local when using --userns=keep-id #23369

Permission denied in .local when using --userns=keep-id #23369

Comments

snyderth commented Jul 23, 2024

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

rhatdan commented Jul 23, 2024

giuseppe commented Jul 23, 2024

snyderth commented Jul 23, 2024

rhatdan commented Jul 23, 2024

snyderth commented Jul 23, 2024

snyderth commented Jul 24, 2024 • edited Loading

rhatdan commented Jul 24, 2024

snyderth commented Jul 24, 2024

rhatdan commented Jul 24, 2024

rhatdan commented Jul 24, 2024

snyderth commented Jul 24, 2024

snyderth commented Jul 24, 2024

rhatdan commented Jul 24, 2024

snyderth commented Jul 24, 2024

snyderth commented Jul 24, 2024

giuseppe commented Jul 24, 2024

snyderth commented Jul 24, 2024

giuseppe commented Jul 24, 2024

snyderth commented Jul 24, 2024

giuseppe commented Jul 24, 2024

snyderth commented Jul 24, 2024

snyderth commented Jul 24, 2024 •

edited

Loading