Trying to reconcile differing volume behavior on Mac and Fedora Linux

[jmontleon]

Issue Description

Describe your issue
We are using a compose file to start some containers. It is available here:
https://github.com/konveyor/kai/blob/main/compose.yaml

Within the containers /podman_compose is being mounted rw so we can write logs.

On Mac we are seeing that the permissions for the directory are and the default user (id 1001) can write to the folder:
drwxr-xr-x. 34 default root 1088 Jul 29 18:33 /podman_compose

Meanwhile on Fedora Linux 40 it is mounted and the default user cannot write to it unless I using podman unshare to adjust permissions first:
drwxr-xr-x. 18 root root 4096 Jul 29 18:04 /podman_compose/

We're not sure why we are seeing different behavior and are trying to understand how to make both behave the same.

Steps to reproduce the issue

Steps to reproduce the issue

1. run podman compose up with the linked compose.yaml on Fedora
2. run podman compose up with the linked compose.yaml on Mac

Describe the results you received

Differing volume permissions

Describe the results you expected

Identical volume permissions

podman info output

Mac podman info --debug output:

host:
  arch: arm64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.75
    systemPercent: 0.11
    userPercent: 0.14
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2046
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.8.11-300.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1777672192
  memTotal: 8298799104
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.11.0-1.20240628130058229856.main.10.g5ad6420.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.0-dev
    package: netavark-1.11.0-1.20240702123536284903.main.32.g49fb0c2.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.0-dev
  ociRuntime:
    name: crun
    package: crun-1.15-1.20240708144150212138.main.51.g6c158dd.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 54f958d21c4e2299eae6b0f4d8b742304540dce6
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240624.g1ee2eca-1.fc40.aarch64
    version: |
      pasta 0^20240624.g1ee2eca-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 225h 17m 9.00s (Approximately 9.38 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 107842875392
  graphRootUsed: 46958817280
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 55
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.2
  Built: 1720569600
  BuiltTime: Tue Jul  9 20:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.5
  Os: linux
  OsArch: linux/arm64
  Version: 5.1.2

Fedora 40 podman info --debug output:

host:
  arch: amd64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 97.55
    systemPercent: 0.15
    userPercent: 2.3
  cpus: 88
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    version: "40"
  eventLogger: journald
  freeLocks: 2043
  hostname: towerofpower.montleon.net
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.9.11-200.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 98757492736
  memTotal: 135042445312
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.11.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.11.0
    package: netavark-1.11.0-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.11.0
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240624.g1ee2eca-1.fc40.x86_64
    version: |
      pasta 0^20240624.g1ee2eca-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 25765601280
  swapTotal: 25765601280
  uptime: 23h 37m 59.00s (Approximately 0.96 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/jason/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 2
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jason/.local/share/containers/storage
  graphRootAllocated: 1981424369664
  graphRootUsed: 85427179520
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/jason/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.2
  Built: 1720569600
  BuiltTime: Tue Jul  9 20:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.5
  Os: linux
  OsArch: linux/amd64
  Version: 5.1.2

### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

No

### Additional environment details

podman machine is being used on Mac because it is required, but not on Fedora.

### Additional information

_No response_

[jmontleon]

Trying with a clean Fedora 40 install and podman without machine the mount was owned by root:root and could not be written to. Trying a second clean install with podman-remote/machine it behaved the same, so it seems this behavior is the same on Fedora regardless of using podman machine.

[rhatdan]

Are you using docker-compose of podman-compose?

Could the ${PWD} be pointing at different directories when run from the Mac versus run from Fedora?

Is ${PWD} in the users homedir? If yes then the file system is shared over virtiofsd on the MAC, whereas it is just a bind mount on Fedora.

[jmontleon]

We are using podman-compose.

When podman-compose is run it is being done from within the git repo cloned in the users home directory, so ${PWD} is somewhere in the home directory on each host.

I tried on two separate clean Fedora 40 VM installs last night, one using podman-remote with a podman machine, one a local connection, and the permissions of the mount in both cases were root:root.

[rhatdan]

Well we don't directly support Podman-compose, but one thing you might try is to add a :U to your volume mount.

       The  :U  suffix  tells  Podman to use the correct host UID and GID based on the UID and GID within the
       container, to change recursively the owner and group of the source volume.  Chowning  walks  the  file
       system  under  the volume and changes the UID/GID on each file. If the volume has thousands of inodes,
       this process takes a long time, delaying the start of the container.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since I never heard back, closing, reopen if I am mistaken.