rootless permissions #20045

InvisibleRasta · 2023-09-19T22:38:01Z

Issue Description

Been trying to use rootless podman the whole day and i always encounter issues with permissions.
podman version
Client: Podman Engine
Version: 4.3.1
API Version: 4.3.1
Go Version: go1.19.8
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64

Steps to reproduce the issue

Steps to reproduce the issue
1.podman run 'docker.io/alpine:latest'
2.
3.

Describe the results you received

podman run 'docker.io/alpine:latest'
Trying to pull docker.io/library/alpine:latest...
Error: initializing destination containers-storage:[vfs@/home/invra/.local/share/containers/storage+/run/user/1000/containers]docker.io/library/alpine:latest: creating a temporary directory: mkdir /var/tmp/storage3641978907: permission denied

Describe the results you expected

run the container

podman info output

podman info                                                                                                                       22:33
host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 95.49
    systemPercent: 1.23
    userPercent: 3.28
  cpus: 12
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  hostname: debian
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.0-12-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 7319117824
  memTotal: 16564797440
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+b1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 2147811328
  swapTotal: 2147811328
  uptime: 0h 34m 56.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.fedoraproject.org
  - docker.io
store:
  configFile: /home/invra/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/invra/.local/share/containers/storage
  graphRootAllocated: 508795289600
  graphRootUsed: 203295784960
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/invra/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

No

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

rhatdan · 2023-09-20T10:07:41Z

First can you configure your storage.conf file to use overlay, not vfs? Are you on Debian?

Second is your homedir local or NFS?

giuseppe · 2023-09-20T11:27:36Z

why is it failing to write to /var/tmp? Does your user have access to that directory? Could you share the output of $ stat -f /var/tmp/?

InvisibleRasta · 2023-09-20T14:37:50Z

yes i am on a clean debian install, how do i configure that storage.conf?
I am on a standard gnome debian desktop installation and my homedir is local
It seems that debian bookworm package does not provide a storage.conf file

stat -f /var/tmp/ 
  File: "/var/tmp/"
    ID: ff9bb48e326e2914 Namelen: 255     Type: btrfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 124217600  Free: 81461395   Available: 80283287
Inodes: Total: 0          Free: 0

rhatdan · 2023-09-20T14:41:43Z

cp /usr/share/containers/storage.conf /etc/containers/storage.conf and then edit the storage.conf

InvisibleRasta · 2023-09-20T14:45:41Z

That file is not provided on debian bookworm package. It looks like that the package that provides storage.conf on debian is containers-storage. Maybe it should be a dependency for podman...

rhatdan · 2023-09-20T15:03:33Z

storage.conf

InvisibleRasta · 2023-09-20T16:09:50Z

Allrigh I have been trying on my fedora install aswell and I allways get the same results. I tried compying the storage.conf on both debian and fedora and no luck starting the container. I also tried to use distrobox and I get the same problem.
distrobox-create --name arch --image archlinux:latest && distrobox enter arch 0.492s 18:06

Image archlinux:latest not found.
Do you want to pull the image now? [Y/n]: 
Resolved "archlinux" as an alias (/home/invra/.cache/containers/short-name-aliases.conf)
Trying to pull docker.io/library/archlinux:latest...
Getting image source signatures
Copying blob 90fae3408070 done  
Copying blob 26c6ccfa2cfa done  
Copying config 3deba8d795 done  
Writing manifest to image destination
3deba8d795ffac7354f286e59018286e406bf3144175fea3a86b116b404ab520
Creating 'arch' using image archlinux:latest	 [ OK ]
Distrobox 'arch' successfully created.
To enter, run:

distrobox enter arch

Container arch is not running.
Starting container arch
run this command to follow along:

 podman logs -f arch

Error: unable to start container "012afa3a863b91eac375f78553aca49b57e3a96579c9ca297c8ca72440e8e424": crun: make `/home/invra/.local/share/containers/storage/overlay/3ef3a38b0ad43168cafda5c33c9fb34e0c8ae9c4d535cc3d80cf968d260bd598/merged` private: Permission denied: OCI permission denied

EDIT: jsut for the scope of testing i tried using docker instead of podman and everythign works with it,

InvisibleRasta · 2023-09-26T05:04:28Z

anyone got any suggestions?

giuseppe · 2023-09-26T15:02:38Z

are you able to run mkdir /var/tmp/storage3641978907 as your unprivileged user or does it fail?

EDIT: jsut for the scope of testing i tried using docker instead of podman and everythign works with it,

are you running rootless Docker?

InvisibleRasta · 2023-09-26T22:59:15Z

mkdir: created directory '/var/tmp/storage3641978907'

and yes rootless docker works

giuseppe · 2023-09-27T14:04:04Z

does it work if you create the container manually without distrobox?

I am not familiar with distrobox, please provide the podman command line that causes that issue, I am still not able to reproduce locally

giuseppe · 2023-09-27T14:10:32Z

can you please run something like podman unshare strace -vv -s1000 -f -o /tmp/strace.log podman run alpine true

and attach here the /tmp/strace.log file you get?

giuseppe · 2023-09-27T14:43:20Z

I've just tried on a fresh Debian 12 machine:

host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 97.28
    systemPercent: 0.93
    userPercent: 1.79
  cpus: 2
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  hostname: debian-s-2vcpu-4gb-120gb-intel-fra1-01
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.0-9-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 3096416256
  memTotal: 4105707520
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+b1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 0h 20m 46.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/foouser/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/foouser/.local/share/containers/storage
  graphRootAllocated: 126647394304
  graphRootUsed: 1509175296
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/foouser/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

and I cannot reproduce the problem. distrobox works as well:

$ distrobox enter arch
Error: inspecting object: no such container arch
Cannot find container arch
Create it now, out of image registry.fedoraproject.org/fedora-toolbox:36? [Y/n]: y
Creating the container with command:
  /usr/bin/distrobox-create -i registry.fedoraproject.org/fedora-toolbox:36 -n arch
Image registry.fedoraproject.org/fedora-toolbox:36 not found.
Do you want to pull the image now? [Y/n]: y
Trying to pull registry.fedoraproject.org/fedora-toolbox:36...
Getting image source signatures
Copying blob 77bc8c6d30dc done  
Copying blob cd7201a4f009 done  
Copying config 5be0bf1ecd done  
Writing manifest to image destination
Storing signatures
5be0bf1ecd38baa4ecf069948fecaf56f1b61cba5738ed2ff7aecbda8b4c0d90
Creating 'arch' using image registry.fedoraproject.org/fedora-toolbox:36         [ OK ]
Distrobox 'arch' successfully created.
To enter, run:

distrobox enter arch

arch
Container arch is not running.
Starting container arch
run this command to follow along:

 podman logs -f arch

 Starting container...                           [ OK ]
 Installing basic packages...                    [ OK ]
 Setting up read-only mounts...                  [ OK ]
 Setting up read-write mounts...                 [ OK ]
 Setting up host's sockets integration...        [ OK ]
 Integrating host's themes, icons, fonts...      [ OK ]
 Setting up host's sockets integration...        [ OK ]
 Integrating host's themes, icons, fonts...      [ OK ]
 Setting up package manager exceptions...        [ OK ]
 Setting up rpm exceptions...                    [ OK ]
 Setting up sudo...                              [ OK ]
 Setting up groups...                            [ OK ]
 Setting up users...                             [ OK ]
 Executing init hooks...                         [ OK ]

Container Setup Complete!

InvisibleRasta · 2023-09-27T23:58:44Z

strace.log
Here is the strace.log you asked me.

ThoriumTextile · 2023-10-16T17:17:50Z

Same issue here, but I get

Error: unable to start container "44a2911cbf75b2c0240d5ec9182f5d803dd022d7e5ac29a3cf3c536fed7458fb": crun: make `/home/user/.local/share/containers/storage/overlay/086e6b4ff3bc061c28d7bdb08200672d27c2ee53c210d6ae7b9fa8c5ecfa33aa/merged` private: Permission denied: OCI permission denied

instead.

github-actions · 2023-11-16T00:06:42Z

A friendly reminder that this issue had no activity for 30 days.

oseiberts11 · 2024-09-03T15:56:14Z

I seem to have the same issue (Ubuntu 22.04) and I haven't found a cause or a fix...

rhatdan · 2024-09-03T15:59:07Z

I would open this as a new issue, I think this issue was either fixed naturally or people moved on.

oseiberts11 · 2024-09-04T07:49:22Z

I don't want to be that user that says "Solved the problem" without saying how, so here is what was the issue in my case. The first podman pull ... was run from a systemd unit with PrivateTmp=yes. Apparently that messed up things enough that even running podman pull normally from the shell was still confused. Removing that from the systemd unit, and podman system reset as this user solved it.

InvisibleRasta added the kind/bug Categorizes issue or PR as related to a bug. label Sep 19, 2023

github-actions bot added the stale-issue label Nov 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rootless permissions #20045

rootless permissions #20045

InvisibleRasta commented Sep 19, 2023

rhatdan commented Sep 20, 2023

giuseppe commented Sep 20, 2023 •

edited

Loading

InvisibleRasta commented Sep 20, 2023 •

edited

Loading

rhatdan commented Sep 20, 2023

InvisibleRasta commented Sep 20, 2023 •

edited

Loading

rhatdan commented Sep 20, 2023

InvisibleRasta commented Sep 20, 2023 •

edited

Loading

InvisibleRasta commented Sep 26, 2023

giuseppe commented Sep 26, 2023

InvisibleRasta commented Sep 26, 2023

giuseppe commented Sep 27, 2023

giuseppe commented Sep 27, 2023

giuseppe commented Sep 27, 2023

InvisibleRasta commented Sep 27, 2023

ThoriumTextile commented Oct 16, 2023

github-actions bot commented Nov 16, 2023

oseiberts11 commented Sep 3, 2024

rhatdan commented Sep 3, 2024

oseiberts11 commented Sep 4, 2024 •

edited

Loading

rootless permissions #20045

rootless permissions #20045

Comments

InvisibleRasta commented Sep 19, 2023

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

rhatdan commented Sep 20, 2023

giuseppe commented Sep 20, 2023 • edited Loading

InvisibleRasta commented Sep 20, 2023 • edited Loading

rhatdan commented Sep 20, 2023

InvisibleRasta commented Sep 20, 2023 • edited Loading

rhatdan commented Sep 20, 2023

InvisibleRasta commented Sep 20, 2023 • edited Loading

InvisibleRasta commented Sep 26, 2023

giuseppe commented Sep 26, 2023

InvisibleRasta commented Sep 26, 2023

giuseppe commented Sep 27, 2023

giuseppe commented Sep 27, 2023

giuseppe commented Sep 27, 2023

InvisibleRasta commented Sep 27, 2023

ThoriumTextile commented Oct 16, 2023

github-actions bot commented Nov 16, 2023

oseiberts11 commented Sep 3, 2024

rhatdan commented Sep 3, 2024

oseiberts11 commented Sep 4, 2024 • edited Loading

giuseppe commented Sep 20, 2023 •

edited

Loading

InvisibleRasta commented Sep 20, 2023 •

edited

Loading

InvisibleRasta commented Sep 20, 2023 •

edited

Loading

InvisibleRasta commented Sep 20, 2023 •

edited

Loading

oseiberts11 commented Sep 4, 2024 •

edited

Loading