Inconsistent full filesystem mount results with xfs vs other e.g. ext4 - can not mount xfs

[mpryc]

Issue Description

Using podman on a fully populated filesystem yields different results with XFS compared to other filesystems like ext4, making it impossible to free up disk space in some scenarios.

This issue is especially troublesome when it propagates to Kubernetes pods trying to mount such XFS filesystems.

Steps to reproduce the issue

Steps to reproduce the issue:

Consider two scripts that gives two inconsistent results, with XFS it gives error while with ext4 it properly mounts the filesystem:

XFS use case:

$ FILESYSTEM=xfs
$ TMP_FILE_PATH="/tmp/myfile_${FILESYSTEM}.dd"
$ dd if=/dev/zero of="${TMP_FILE_PATH}" bs=4k iflag=fullblock,count_bytes count=100M
$ mkfs -t "${FILESYSTEM}" "${TMP_FILE_PATH}"
$ mkdir "/tmp/testmount_${FILESYSTEM}"
$ sudo mount -t "${FILESYSTEM}" -o loop "${TMP_FILE_PATH}" "/tmp/testmount_${FILESYSTEM}"
$ sudo dd if=/dev/random of="/tmp/testmount_${FILESYSTEM}/full_filesystem.abc"
$ sudo podman run --name=test"${FILESYSTEM}" --volume "/tmp/testmount_${FILESYSTEM}":"/mnt/test${FILESYSTEM}":Z --rm -it registry.access.redhat.com/ubi9/ubi /bin/bash
Error: lsetxattr /tmp/testmount_xfs: no space left on device

EXT4 use case:

$ FILESYSTEM=ext4
$ TMP_FILE_PATH="/tmp/myfile_${FILESYSTEM}.dd"
$ dd if=/dev/zero of="${TMP_FILE_PATH}" bs=4k iflag=fullblock,count_bytes count=100M
$ mkfs -t "${FILESYSTEM}" "${TMP_FILE_PATH}"
$ mkdir "/tmp/testmount_${FILESYSTEM}"
$ sudo mount -t "${FILESYSTEM}" -o loop "${TMP_FILE_PATH}" "/tmp/testmount_${FILESYSTEM}"
$ sudo dd if=/dev/random of="/tmp/testmount_${FILESYSTEM}/full_filesystem.abc"
$ sudo podman run --name=test"${FILESYSTEM}" --volume "/tmp/testmount_${FILESYSTEM}":"/mnt/test${FILESYSTEM}":Z --rm -it registry.access.redhat.com/ubi9/ubi /bin/bash
[root@e778ab568f3b /]#

Describe the results you received

With XFS filesystem:

Error: lsetxattr /tmp/testmount_xfs: no space left on device

With ext4 filesystem, the filesystem was mounted and the container started allowing to manipulate or free up space:

[root@e778ab568f3b /]#

Describe the results you expected

Both cases should be consistent regardless if it's an error or proper container run.

podman info output

Note, this is observed as well with much newer versions of podman, here is one on local laptop.


$ podman info
host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 92.25
    systemPercent: 1.43
    userPercent: 6.32
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "37"
  eventLogger: journald
  freeLocks: 2004
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.4.6-100.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1760374784
  memTotal: 33335476224
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc37.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc37.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230625.g32660ce-1.fc37.x86_64
    version: |
      pasta 0^20230625.g32660ce-1.fc37.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 6604189696
  swapTotal: 8589930496
  uptime: 215h 50m 12.00s (Approximately 8.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/migi/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/migi/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 409521610752
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 253
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/migi/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1693251511
  BuiltTime: Mon Aug 28 21:38:31 2023
  GitCommit: ""
  GoVersion: go1.19.12
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

I don't see what this has to do with podman at all?

If xfs fails on lsetxattr with no space left on device then this is nothing podman can ever fix.
And if the syscall works on ext4 then podman thinks it works of course. It is practically impossible for podman to catch this.

[Luap99]

And if you do not want to use lsetxattr then remove the :Z and run the container with selinux disabled

[mpryc]

@Luap99 I logged this against podman because it's the podman CLI that behaves differently for the user. If you feel like it's an underlying library please recommend where it should be transferred. The bug is a result against tooling that was running in Kubernetes cluster which is using podman as backend. The tooling can't fix the issue as the problem is much lower in the stack.

From the higher level tooling one recommendation is to actually not use xfs (which is really wrong thing to recommend) or to expand xfs to clean up the space before running pod with attached xfs filesystem.

[Luap99]

:Z tells us to relabel the volume (lsetxattr), whenever that syscall works on a full fs or not is not up to us and we either get a error from the kernel that we should report to the user or we do not get a error in which case it works of course.
You need to talk to kernel fs developers and ask them why works on one fs and not the other

[mpryc]

@Luap99 right, so there may be a bug in how the kernel behaves on different fs in this case. For me it should not matter which filesystem is being used as long as it's consistent behavior between them. Thank you !

[Luap99]

Then do not use :Z

[mpryc]

Just for reference, it's possible to set the security context using ext4 partition type, however the xfs gives error

$ sudo chcon -Rt svirt_sandbox_file_t /tmp/testmount_ext4/
$ echo $?
0

$ sudo chcon -Rt svirt_sandbox_file_t /tmp/testmount_xfs/
chcon: failed to change context of 'full_filesystem.abc' to ‘unconfined_u:object_r:svirt_sandbox_file_t:s0’: No space left on device
chcon: failed to change context of '/tmp/testmount_xfs/' to ‘system_u:object_r:svirt_sandbox_file_t:s0’: No space left on device
$ echo $?
1

[rhatdan]

I believe this is a known xfs issue, but should be opened against the kernel, not podman.