rootless buildah setfattr operation not supported

[llegolas]

Trying to use buildah from scratch on recent fedora 41 I've stumbled with a problem when truing to install packages with dnf.
DNF was failing with rpm unpacking errors which I've tracked down to rpm-plugin-ima. In short it seem overlay storage driver is not allowing setting extended file attributes. Here are the steps to reproduce:

$ buildah unshare
# ctr=$(buildah from scratch)
# mnt=$(buildah mount $ctr)
# echo $ctr $mnt
working-container /home/XXXXX/.local/share/containers/storage/overlay/1b35fea4a195206e1a34f93b6734bc2792c50dbf65aff93e378575286fa80b6e/merged
# dnf install --installroot $mnt --nodocs --releasever 41 --setopt=install_weak_deps=false coreutils bash --use-host-config -y
Updating and loading repositories:
 Fedora 41 - x86_64 - Updates                                                                                                                         100% |   2.3 MiB/s |   4.2 MiB |  00m02s
 Copr repo for libfprint-tod-goodix owned by manciukic                                                                                                100% |  14.1 KiB/s |   3.8 KiB |  00m00s
 Fedora 41 openh264 (From Cisco) - x86_64                                                                                                             100% |   2.9 KiB/s |   4.8 KiB |  00m02s
 Fedora 41 - x86_64                                                                                                                                   100% |   5.7 MiB/s |  35.4 MiB |  00m06s
 RPM Fusion for Fedora 41 - Free tainted                                                                                                              100% |  18.8 KiB/s |  13.1 KiB |  00m01s
 Microsoft Teams                                                                                                                                      100% |   5.5 KiB/s |   1.6 KiB |  00m00s
 RPM Fusion for Fedora 41 - Free - Updates                                                                                                            100% |  32.2 KiB/s |  18.0 KiB |  00m01s
 RPM Fusion for Fedora 41 - Free                                                                                                                      100% | 441.0 KiB/s | 170.2 KiB |  00m00s
 RPM Fusion for Fedora 41 - Nonfree tainted                                                                                                           100% |  18.2 KiB/s |  12.4 KiB |  00m01s
 slack                                                                                                                                                100% |   1.7 KiB/s |   4.5 KiB |  00m03s
 RPM Fusion for Fedora 41 - Nonfree - Updates                                                                                                         100% |  11.0 KiB/s |  17.2 KiB |  00m02s
 RPM Fusion for Fedora 41 - Nonfree                                                                                                                   100% | 104.1 KiB/s |  85.8 KiB |  00m01s
 teams                                                                                                                                                100% |  11.7 KiB/s |   1.6 KiB |  00m00s
 vscodium                                                                                                                                             100% |  18.6 KiB/s |   3.8 KiB |  00m00s
>>> Librepo error: repomd.xml GPG signature verification error: Signing key not found
 https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/-/raw/master/pub.gpg                                                                            100% |  11.3 KiB/s |   3.1 KiB |  00m00sImporting PGP key 0x5A278D9C:
 UserID     : "Pavlo Rudyi <[email protected]>"
 Fingerprint: 1302DE60231889FE1EBACADC54678CF75A278D9C
 From       : https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/-/raw/master/pub.gpg
The key was successfully imported.

 vscodium                                                                                                                                             100% |  19.5 KiB/s |   6.1 KiB |  00m00s
Repositories loaded.
Package                                                          Arch           Version                                                          Repository                               Size
Installing:
 bash                                                            x86_64         5.2.32-1.fc41                                                    fedora                                8.2 MiB
 coreutils                                                       x86_64         9.5-10.fc41                                                      fedora                                5.6 MiB
Installing dependencies:
 alternatives                                                    x86_64         1.30-1.fc41                                                      fedora                               66.3 KiB
 ...
 ...
 zlib-ng-compat                                                  x86_64         2.1.7-3.fc41                                                     fedora                              134.0 KiB

Transaction Summary:
 Installing:        40 packages

Total size of inbound packages is 16 MiB. Need to download 16 MiB.
After this operation, 54 MiB extra will be used (install 54 MiB, remove 0 B).
^CFailed to download packages
 Librepo error: Interrupted by a SIGINT signal
 # dnf install --installroot $mnt --nodocs --releasever 41 --setopt=install_weak_deps=false coreutils bash --use-host-config -y
Updating and loading repositories:
Repositories loaded.
Package                                                          Arch           Version                                                          Repository                               Size
Installing:
 bash                                                            x86_64         5.2.32-1.fc41                                                    fedora                                8.2 MiB
 coreutils                                                       x86_64         9.5-10.fc41                                                      fedora                                5.6 MiB
...
...

Transaction Summary:
 Installing:        40 packages

Total size of inbound packages is 16 MiB. Need to download 16 MiB.
After this operation, 54 MiB extra will be used (install 54 MiB, remove 0 B).
[ 1/40] gmp-1:6.3.0-2.fc41.x86_64                                                                                                                     100% |   3.8 MiB/s | 318.0 KiB |  00m00s
...
...
[40/40] p11-kit-trust-0:0.25.5-3.fc41.x86_64                                                                                                          100% | 444.9 KiB/s | 132.1 KiB |  00m00s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[40/40] Total                                                                                                                                         100% |   4.8 MiB/s |  15.9 MiB |  00m03s
Running transaction
Importing PGP key 0xE99D6AD1:
 UserID     : "Fedora (41) <[email protected]>"
 Fingerprint: 466CF2D8B60BC3057AA9453ED0622462E99D6AD1
 From       : file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-41-x86_64
The key was successfully imported.
[ 1/42] Verify package files                                                                                                                          100% | 232.0   B/s |  40.0   B |  00m00s
[ 2/42] Prepare transaction                                                                                                                           100% | 975.0   B/s |  40.0   B |  00m00s
[ 3/42] Installing libgcc-0:14.2.1-3.fc41.x86_64                                                                                                      100% |  20.8 MiB/s | 276.3 KiB |  00m00s
>>> Unpack error: libgcc-0:14.2.1-3.fc41.x86_64
[ 1/42] Installing fedora-release-identity-basic-0:41-27.noarch                                                                                       100% |   0.0   B/s | 940.0   B |  00m00s
>>> Unpack error: fedora-release-identity-basic-0:41-27.noarch
[ 1/42] Installing fedora-gpg-keys-0:41-1.noarch                                                                                                      100% |  24.0 MiB/s | 172.2 KiB |  00m00s
>>> Unpack error: fedora-gpg-keys-0:41-1.noarch
[ 1/42] Installing fedora-repos-0:41-1.noarch                                                                                                         100% |   2.8 MiB/s |   5.7 KiB |  00m00s
[ 2/42] Installing fedora-release-common-0:41-27.noarch                                                                                               100% |   7.8 MiB/s |  23.9 KiB |  00m00s
>>> Unpack error: fedora-release-common-0:41-27.noarch
...
...
[ 1/42] Installing coreutils-0:9.5-10.fc41.x86_64                                                                                                     100% | 808.7 MiB/s |   5.7 MiB |  00m00s
>>> Unpack error: coreutils-0:9.5-10.fc41.x86_64

Transaction failed: Rpm transaction failed.

trying to use rpm directly spat out the ima related rpm error

pm -iv --root $mnt  $mnt/var/cache/libdnf5/updates-e19adde8fd271134/packages/glibc-2.40-9.fc41.x86_64.rpm $mnt/var/cache/libdnf5/updates-e19adde8fd271134/packages/glibc-common-2.40-9.fc41.x86_64.rpm $mnt/var/cache/libdnf5/fedora-7efbab3c1dbcd0d4/packages/bash-5.2.32-1.fc41.x86_64.rpm $mnt/var/cache/libdnf5/fedora-7efbab3c1dbcd0d4/packages/ncurses-libs-6.5-2.20240629.fc41.x86_64.rpm $mnt/var/cache/libdnf5/fedora-7efbab3c1dbcd0d4/packages/ncurses-base-6.5-2.20240629.fc41.noarch.rpm $mnt/var/cache/libdnf5/fedora-7efbab3c1dbcd0d4/packages/libgcc-14.2.1-3.fc41.x86_64.rpm 
Verifying packages...
warning: Unable to get systemd shutdown inhibition lock: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Preparing packages...
libgcc-14.2.1-3.fc41.x86_64
error: ima: could not apply signature on '/lib64/libgcc_s-14-20240912.so.1;672e8d8f': Operation not permitted
error: Plugin ima: hook fsm_file_prepare failed
error: unpacking of archive failed on file /lib64/libgcc_s-14-20240912.so.1;672e8d8f: cpio: (error 0x2)
error: libgcc-14.2.1-3.fc41.x86_64: install failed
ncurses-base-6.5-2.20240629.fc41.noarch
error: ima: could not apply signature on '/usr/share/doc/ncurses-base/README;672e8d8f': Operation not permitted
error: Plugin ima: hook fsm_file_prepare failed
error: unpacking of archive failed on file /usr/share/doc/ncurses-base/README;672e8d8f: cpio: (error 0x2)
error: ncurses-base-6.5-2.20240629.fc41.noarch: install failed
glibc-common-2.40-9.fc41.x86_64
error: ima: could not apply signature on '/usr/bin/gencat;672e8d8f': Operation not permitted
error: Plugin ima: hook fsm_file_prepare failed
error: unpacking of archive failed on file /usr/bin/gencat;672e8d8f: cpio: (error 0x2)
error: glibc-common-2.40-9.fc41.x86_64: install failed
glibc-2.40-9.fc41.x86_64
error: ima: could not apply signature on '/usr/lib64/audit/sotruss-lib.so;672e8d8f': Operation not permitted
error: Plugin ima: hook fsm_file_prepare failed
error: unpacking of archive failed on file /usr/lib64/audit/sotruss-lib.so;672e8d8f: cpio: (error 0x2)
error: glibc-2.40-9.fc41.x86_64: install failed
ncurses-libs-6.5-2.20240629.fc41.x86_64
error: ima: could not apply signature on '/usr/lib64/libform.so.6.5;672e8d8f': Operation not permitted
error: Plugin ima: hook fsm_file_prepare failed
error: unpacking of archive failed on file /usr/lib64/libform.so.6.5;672e8d8f: cpio: (error 0x2)
error: ncurses-libs-6.5-2.20240629.fc41.x86_64: install failed
bash-5.2.32-1.fc41.x86_64
error: ima: could not apply signature on '/usr/bin/alias;672e8d8f': Operation not permitted
error: Plugin ima: hook fsm_file_prepare failed
error: unpacking of archive failed on file /usr/bin/alias;672e8d8f: cpio: (error 0x2)
error: bash-5.2.32-1.fc41.x86_64: install failed

which I was able to workaround by adding --undefine=__transaction_ima to rpm command

rpm -iv --undefine=__transaction_ima --root $mnt  $mnt/var/cache/libdnf5/updates-e19adde8fd271134/packages/glibc-2.40-9.fc41.x86_64.rpm $mnt/var/cache/libdnf5/updates-e19adde8fd271134/packages/glibc-common-2.40-9.fc41.x86_64.rpm $mnt/var/cache/libdnf5/fedora-7efbab3c1dbcd0d4/packages/bash-5.2.32-1.fc41.x86_64.rpm $mnt/var/cache/libdnf5/fedora-7efbab3c1dbcd0d4/packages/ncurses-libs-6.5-2.20240629.fc41.x86_64.rpm $mnt/var/cache/libdnf5/fedora-7efbab3c1dbcd0d4/packages/ncurses-base-6.5-2.20240629.fc41.noarch.rpm $mnt/var/cache/libdnf5/fedora-7efbab3c1dbcd0d4/packages/libgcc-14.2.1-3.fc41.x86_64.rpm
Verifying packages...
warning: Unable to get systemd shutdown inhibition lock: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Preparing packages...
libgcc-14.2.1-3.fc41.x86_64
ncurses-base-6.5-2.20240629.fc41.noarch
glibc-common-2.40-9.fc41.x86_64
glibc-2.40-9.fc41.x86_64
ncurses-libs-6.5-2.20240629.fc41.x86_64
bash-5.2.32-1.fc41.x86_64

I vaguely remember similar problem years ago but it was related to selinux (it stores its labels as xattrs if my memory serves me right too) and I was able to workaround it with dnf .... --setopt tsflags=nocontexts ..... unfortunately there is no tsflag for rpm-ima.

I can provide more info about the system if need be.

[llegolas]

I find this 5e82f27 which seem to try to fix exactly what i observe.
my buildah version is:

$ buildah --version
buildah version 1.37.5 (image-spec 1.1.0, runtime-spec 1.2.0)

so I run the latest release.
@mheon Any ideas ?

[german-rios-gonzalez]

Same happens with version 1.38.0

[mheon]

Is this a RUN statement trying to do IMA things? If so, there's not much we can do - the program is seeing root and assuming it can do any IMA operation when in reality it's a rootless build and it can't. Might have better luck reporting against RPM, which (if it was user namespace aware) could not try the IMA operations at all given it knows they will fail?