RHEL Packages: GPG check fails when explicitly enabled

[ivanov17]

Hello team,

I'm trying to install packages from the Unit repository, but I'm getting the following error:

error: package unit-1.32.1-1.el9.ngx.x86_64 cannot be verified and repo unit is GPG enabled: /var/cache/yum/metadata/unit-9-x86_64/packages/unit-1.32.1-1.el9.ngx.x86_64.rpm could not be verified.
/var/cache/yum/metadata/unit-9-x86_64/packages/unit-1.32.1-1.el9.ngx.x86_64.rpm:  digest:  SIGNATURE:  NOT OK

I have explicitly configured the GPG check. Here is my repository configuration:

[unit]
name=unit repo
baseurl=https://packages.nginx.org/unit/rhel/$releasever/$basearch/
gpgcheck=1
enabled=1
repo_gpgcheck=1
gpgkey=https://nginx.org/keys/nginx_signing.key

I'm not sure when exactly it broke, but I know it worked before. I think this may be related to the GPG key update in early June 2024.

Please fix this. Thank you.

[callahad]

@thresheek Could you please look at this? Thank you 🙏

[thresheek]

Hi @ivanov17!

Did you try to remove and re-import the gpg keys as per the blog guidelines?

[ivanov17]

Hello @thresheek!

I'm building custom container images using the official CentOS Stream 9 base image and packages from the Unit repository. I don't have to re-import the gpg key since a new key is used each time I build.

[thresheek]

Thanks for the details.

It looks like dnf's implementation for repo_gpgcheck= isnt as robust as one would expect (and suprisingly differs from gpgcheck= implementation!) and fails with multiple keys in one file.

There is even a bug from 2019 on RedHat bugzilla about it: https://bugzilla.redhat.com/show_bug.cgi?id=1768206

I think for now your only solution would be to disable repo_gpgcheck and rely on packages signatures instead.

[ivanov17]

Thank you, but it doesn't seem to work. I can only create an image if I disable both options (for repo and packages).

[thresheek]

Can't reproduce with:

$ cat Dockerfile.norepocheck
FROM quay.io/centos/centos:stream9

COPY ./unit-norepocheck.repo /etc/yum.repos.d/unit.repo

RUN set -x \
    && dnf makecache \
    && dnf install -y unit
$ cat unit-norepocheck.repo
[unit]
name=unit repo
baseurl=https://packages.nginx.org/unit/rhel/$releasever/$basearch/
gpgcheck=1
enabled=1
repo_gpgcheck=0
gpgkey=https://nginx.org/keys/nginx_signing.key

$ docker build --progress=plain --no-cache -f Dockerfile.norepocheck .
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile.norepocheck
#1 transferring dockerfile: 207B done
#1 DONE 0.0s

#2 [internal] load metadata for quay.io/centos/centos:stream9
#2 DONE 0.1s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [1/3] FROM quay.io/centos/centos:stream9@sha256:8edcfab3ba262a926f3f911d5743bd894dce857fc80f74b615b68da3d05f4bde
#4 CACHED

#5 [internal] load build context
#5 transferring context: 225B done
#5 DONE 0.0s

#6 [2/3] COPY ./unit-norepocheck.repo /etc/yum.repos.d/unit.repo
#6 DONE 0.0s

#7 [3/3] RUN set -x     && dnf makecache     && dnf install -y unit
#7 0.227 + dnf makecache
#7 1.160 CentOS Stream 9 - BaseOS                         11 MB/s | 8.1 MB     00:00
#7 3.715 CentOS Stream 9 - AppStream                      20 MB/s |  20 MB     00:00
#7 9.505 CentOS Stream 9 - Extras packages                32 kB/s |  17 kB     00:00
#7 9.764 unit repo                                       447 kB/s |  33 kB     00:00
#7 10.85 Metadata cache created.
#7 10.88 + dnf install -y unit
#7 11.24 Last metadata expiration check: 0:00:01 ago on Tue Jun 25 20:13:27 2024.
#7 11.36 Dependencies resolved.
#7 11.36 ================================================================================
#7 11.36  Package                 Arch        Version                  Repository   Size
#7 11.36 ================================================================================
#7 11.36 Installing:
#7 11.36  unit                    x86_64      1.32.1-1.el9.ngx         unit        723 k
#7 11.36 Installing dependencies:
#7 11.36  acl                     x86_64      2.3.1-4.el9              baseos       71 k
#7 11.36  cracklib                x86_64      2.9.6-27.el9             baseos       94 k
#7 11.36  cracklib-dicts          x86_64      2.9.6-27.el9             baseos      3.6 M
#7 11.36  dbus                    x86_64      1:1.12.20-8.el9          baseos      3.8 k
#7 11.36  dbus-broker             x86_64      28-7.el9                 baseos      172 k
#7 11.36  dbus-common             noarch      1:1.12.20-8.el9          baseos       15 k
#7 11.36  diffutils               x86_64      3.7-12.el9               baseos      397 k
#7 11.36  kmod-libs               x86_64      28-9.el9                 baseos       64 k
#7 11.36  libdb                   x86_64      5.3.28-54.el9            baseos      735 k
#7 11.36  libeconf                x86_64      0.4.1-4.el9              baseos       27 k
#7 11.36  libfdisk                x86_64      2.37.4-18.el9            baseos      155 k
#7 11.36  libpwquality            x86_64      1.4.4-8.el9              baseos      119 k
#7 11.36  libseccomp              x86_64      2.5.2-2.el9              baseos       72 k
#7 11.36  libselinux-utils        x86_64      3.6-1.el9                baseos      190 k
#7 11.36  libutempter             x86_64      1.2.1-6.el9              baseos       27 k
#7 11.36  openssl                 x86_64      1:3.2.2-2.el9            baseos      1.4 M
#7 11.36  pam                     x86_64      1.5.1-20.el9             baseos      628 k
#7 11.36  policycoreutils         x86_64      3.6-2.1.el9              baseos      242 k
#7 11.36  systemd                 x86_64      252-37.el9               baseos      4.2 M
#7 11.36  systemd-libs            x86_64      252-37.el9               baseos      680 k
#7 11.36  systemd-pam             x86_64      252-37.el9               baseos      287 k
#7 11.36  systemd-rpm-macros      noarch      252-37.el9               baseos       75 k
#7 11.36  util-linux              x86_64      2.37.4-18.el9            baseos      2.3 M
#7 11.36  util-linux-core         x86_64      2.37.4-18.el9            baseos      465 k
#7 11.36
#7 11.36 Transaction Summary
#7 11.36 ================================================================================
#7 11.36 Install  25 Packages
#7 11.36
#7 11.36 Total download size: 17 M
#7 11.36 Installed size: 49 M
#7 11.36 Downloading Packages:
#7 11.79 (1/25): acl-2.3.1-4.el9.x86_64.rpm              1.7 MB/s |  71 kB     00:00
#7 11.79 (2/25): cracklib-2.9.6-27.el9.x86_64.rpm        2.0 MB/s |  94 kB     00:00
#7 11.80 (3/25): dbus-1.12.20-8.el9.x86_64.rpm           318 kB/s | 3.8 kB     00:00
#7 11.81 (4/25): dbus-common-1.12.20-8.el9.noarch.rpm    1.2 MB/s |  15 kB     00:00
#7 11.82 (5/25): dbus-broker-28-7.el9.x86_64.rpm         6.6 MB/s | 172 kB     00:00
#7 11.83 (6/25): kmod-libs-28-9.el9.x86_64.rpm           4.6 MB/s |  64 kB     00:00
#7 11.85 (7/25): diffutils-3.7-12.el9.x86_64.rpm          11 MB/s | 397 kB     00:00
#7 11.88 (8/25): cracklib-dicts-2.9.6-27.el9.x86_64.rpm   27 MB/s | 3.6 MB     00:00
#7 11.90 (9/25): libeconf-0.4.1-4.el9.x86_64.rpm         554 kB/s |  27 kB     00:00
#7 11.91 (10/25): libdb-5.3.28-54.el9.x86_64.rpm         9.6 MB/s | 735 kB     00:00
#7 11.91 (11/25): libfdisk-2.37.4-18.el9.x86_64.rpm      5.2 MB/s | 155 kB     00:00
#7 11.92 (12/25): libpwquality-1.4.4-8.el9.x86_64.rpm    6.2 MB/s | 119 kB     00:00
#7 11.93 (13/25): libseccomp-2.5.2-2.el9.x86_64.rpm      5.3 MB/s |  72 kB     00:00
#7 11.93 (14/25): libutempter-1.2.1-6.el9.x86_64.rpm     2.0 MB/s |  27 kB     00:00
#7 11.94 (15/25): libselinux-utils-3.6-1.el9.x86_64.rpm  8.0 MB/s | 190 kB     00:00
#7 11.95 (16/25): policycoreutils-3.6-2.1.el9.x86_64.rpm  14 MB/s | 242 kB     00:00
#7 11.96 (17/25): pam-1.5.1-20.el9.x86_64.rpm             20 MB/s | 628 kB     00:00
#7 11.99 (18/25): openssl-3.2.2-2.el9.x86_64.rpm          22 MB/s | 1.4 MB     00:00
#7 12.00 (19/25): systemd-libs-252-37.el9.x86_64.rpm      18 MB/s | 680 kB     00:00
#7 12.01 (20/25): systemd-pam-252-37.el9.x86_64.rpm       16 MB/s | 287 kB     00:00
#7 12.01 (21/25): systemd-rpm-macros-252-37.el9.noarch.r 5.7 MB/s |  75 kB     00:00
#7 12.06 (22/25): systemd-252-37.el9.x86_64.rpm           40 MB/s | 4.2 MB     00:00
#7 12.07 (23/25): util-linux-core-2.37.4-18.el9.x86_64.r 8.5 MB/s | 465 kB     00:00
#7 12.10 (24/25): util-linux-2.37.4-18.el9.x86_64.rpm     24 MB/s | 2.3 MB     00:00
#7 12.11 (25/25): unit-1.32.1-1.el9.ngx.x86_64.rpm        13 MB/s | 723 kB     00:00
#7 12.12 --------------------------------------------------------------------------------
#7 12.12 Total                                            22 MB/s |  17 MB     00:00
#7 12.13 CentOS Stream 9 - BaseOS                        1.6 MB/s | 1.6 kB     00:00
#7 12.19 Importing GPG key 0x8483C65D:
#7 12.19  Userid     : "CentOS (CentOS Official Signing Key) <[email protected]>"
#7 12.19  Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
#7 12.19  From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
#7 12.24 Key imported successfully
#7 12.72 unit repo                                       501 kB/s |  12 kB     00:00
#7 12.78 Importing GPG key 0xB49F6B46:
#7 12.78  Userid     : "nginx signing key <[email protected]>"
#7 12.78  Fingerprint: 8540 A6F1 8833 A80E 9C16 53A4 2FD2 1310 B49F 6B46
#7 12.78  From       : https://nginx.org/keys/nginx_signing.key
#7 12.78 Key imported successfully
#7 12.78 Importing GPG key 0x7BD9BF62:
#7 12.78  Userid     : "nginx signing key <[email protected]>"
#7 12.78  Fingerprint: 573B FD6B 3D8F BC64 1079 A6AB ABF5 BD82 7BD9 BF62
#7 12.78  From       : https://nginx.org/keys/nginx_signing.key
#7 12.79 Key imported successfully
#7 12.79 Importing GPG key 0x8D88A2B3:
#7 12.79  Userid     : "nginx signing key <[email protected]>"
#7 12.79  Fingerprint: 9E9B E90E ACBC DE69 FE9B 204C BCDC D8A3 8D88 A2B3
#7 12.79  From       : https://nginx.org/keys/nginx_signing.key
#7 12.79 Key imported successfully
#7 12.81 Running transaction check
#7 12.87 Transaction check succeeded.
#7 12.87 Running transaction test
#7 13.17 Transaction test succeeded.
#7 13.17 Running transaction
#7 13.58   Preparing        :                                                        1/1
#7 13.63   Installing       : systemd-libs-252-37.el9.x86_64                        1/25
#7 13.64   Running scriptlet: systemd-libs-252-37.el9.x86_64                        1/25
#7 13.69   Installing       : libselinux-utils-3.6-1.el9.x86_64                     2/25
#7 13.70   Installing       : libfdisk-2.37.4-18.el9.x86_64                         3/25
#7 13.71   Installing       : cracklib-2.9.6-27.el9.x86_64                          4/25
#7 13.80   Installing       : cracklib-dicts-2.9.6-27.el9.x86_64                    5/25
#7 13.84   Installing       : util-linux-core-2.37.4-18.el9.x86_64                  6/25
#7 13.84   Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64                  6/25
#7 13.85   Installing       : systemd-rpm-macros-252-37.el9.noarch                  7/25
#7 13.91   Installing       : openssl-1:3.2.2-2.el9.x86_64                          8/25
#7 13.92   Running scriptlet: libutempter-1.2.1-6.el9.x86_64                        9/25
#7 13.98   Installing       : libutempter-1.2.1-6.el9.x86_64                        9/25
#7 13.99   Installing       : libseccomp-2.5.2-2.el9.x86_64                        10/25
#7 14.00   Installing       : libeconf-0.4.1-4.el9.x86_64                          11/25
#7 14.02   Installing       : libdb-5.3.28-54.el9.x86_64                           12/25
#7 14.07   Installing       : pam-1.5.1-20.el9.x86_64                              13/25
#7 14.09   Installing       : libpwquality-1.4.4-8.el9.x86_64                      14/25
#7 14.19   Installing       : util-linux-2.37.4-18.el9.x86_64                      15/25
#7 14.19 warning: /etc/adjtime created as /etc/adjtime.rpmnew
#7 14.19
#7 14.21   Installing       : kmod-libs-28-9.el9.x86_64                            16/25
#7 14.23   Installing       : diffutils-3.7-12.el9.x86_64                          17/25
#7 14.25   Installing       : policycoreutils-3.6-2.1.el9.x86_64                   18/25
#7 14.26   Running scriptlet: policycoreutils-3.6-2.1.el9.x86_64                   18/25
#7 14.27   Installing       : acl-2.3.1-4.el9.x86_64                               19/25
#7 14.28   Installing       : dbus-1:1.12.20-8.el9.x86_64                          20/25
#7 14.29   Installing       : systemd-pam-252-37.el9.x86_64                        21/25
#7 14.29   Running scriptlet: systemd-252-37.el9.x86_64                            22/25
#7 14.64   Installing       : systemd-252-37.el9.x86_64                            22/25
#7 14.66   Running scriptlet: systemd-252-37.el9.x86_64                            22/25
#7 14.85   Installing       : dbus-common-1:1.12.20-8.el9.noarch                   23/25
#7 14.85   Running scriptlet: dbus-common-1:1.12.20-8.el9.noarch                   23/25
#7 14.88 Created symlink /etc/systemd/system/sockets.target.wants/dbus.socket → /usr/lib/systemd/system/dbus.socket.
#7 14.88 Created symlink /etc/systemd/user/sockets.target.wants/dbus.socket → /usr/lib/systemd/user/dbus.socket.
#7 14.88
#7 14.88   Running scriptlet: dbus-broker-28-7.el9.x86_64                          24/25
#7 14.95   Installing       : dbus-broker-28-7.el9.x86_64                          24/25
#7 14.96   Running scriptlet: dbus-broker-28-7.el9.x86_64                          24/25
#7 14.98 Created symlink /etc/systemd/system/dbus.service → /usr/lib/systemd/system/dbus-broker.service.
#7 14.98 Created symlink /etc/systemd/user/dbus.service → /usr/lib/systemd/user/dbus-broker.service.
#7 14.98
#7 15.01   Installing       : unit-1.32.1-1.el9.ngx.x86_64                         25/25
#7 15.01   Running scriptlet: unit-1.32.1-1.el9.ngx.x86_64                         25/25
#7 15.09 ----------------------------------------------------------------------
#7 15.09
#7 15.09 Thank you for installing NGINX Unit!
#7 15.09
#7 15.09 Additional modules are available in standalone packages.
#7 15.09 To see the available modules, run "yum list available unit-\*".
#7 15.09
#7 15.09 Online documentation is available at https://unit.nginx.org/
#7 15.09
#7 15.09 ----------------------------------------------------------------------
#7 15.09
#7 15.35   Verifying        : acl-2.3.1-4.el9.x86_64                                1/25
#7 15.35   Verifying        : cracklib-2.9.6-27.el9.x86_64                          2/25
#7 15.35   Verifying        : cracklib-dicts-2.9.6-27.el9.x86_64                    3/25
#7 15.35   Verifying        : dbus-1:1.12.20-8.el9.x86_64                           4/25
#7 15.35   Verifying        : dbus-broker-28-7.el9.x86_64                           5/25
#7 15.35   Verifying        : dbus-common-1:1.12.20-8.el9.noarch                    6/25
#7 15.35   Verifying        : diffutils-3.7-12.el9.x86_64                           7/25
#7 15.35   Verifying        : kmod-libs-28-9.el9.x86_64                             8/25
#7 15.35   Verifying        : libdb-5.3.28-54.el9.x86_64                            9/25
#7 15.35   Verifying        : libeconf-0.4.1-4.el9.x86_64                          10/25
#7 15.35   Verifying        : libfdisk-2.37.4-18.el9.x86_64                        11/25
#7 15.35   Verifying        : libpwquality-1.4.4-8.el9.x86_64                      12/25
#7 15.35   Verifying        : libseccomp-2.5.2-2.el9.x86_64                        13/25
#7 15.35   Verifying        : libselinux-utils-3.6-1.el9.x86_64                    14/25
#7 15.35   Verifying        : libutempter-1.2.1-6.el9.x86_64                       15/25
#7 15.35   Verifying        : openssl-1:3.2.2-2.el9.x86_64                         16/25
#7 15.35   Verifying        : pam-1.5.1-20.el9.x86_64                              17/25
#7 15.35   Verifying        : policycoreutils-3.6-2.1.el9.x86_64                   18/25
#7 15.36   Verifying        : systemd-252-37.el9.x86_64                            19/25
#7 15.36   Verifying        : systemd-libs-252-37.el9.x86_64                       20/25
#7 15.36   Verifying        : systemd-pam-252-37.el9.x86_64                        21/25
#7 15.36   Verifying        : systemd-rpm-macros-252-37.el9.noarch                 22/25
#7 15.36   Verifying        : util-linux-2.37.4-18.el9.x86_64                      23/25
#7 15.36   Verifying        : util-linux-core-2.37.4-18.el9.x86_64                 24/25
#7 15.36   Verifying        : unit-1.32.1-1.el9.ngx.x86_64                         25/25
#7 15.47
#7 15.47 Installed:
#7 15.47   acl-2.3.1-4.el9.x86_64                 cracklib-2.9.6-27.el9.x86_64
#7 15.47   cracklib-dicts-2.9.6-27.el9.x86_64     dbus-1:1.12.20-8.el9.x86_64
#7 15.47   dbus-broker-28-7.el9.x86_64            dbus-common-1:1.12.20-8.el9.noarch
#7 15.47   diffutils-3.7-12.el9.x86_64            kmod-libs-28-9.el9.x86_64
#7 15.47   libdb-5.3.28-54.el9.x86_64             libeconf-0.4.1-4.el9.x86_64
#7 15.47   libfdisk-2.37.4-18.el9.x86_64          libpwquality-1.4.4-8.el9.x86_64
#7 15.47   libseccomp-2.5.2-2.el9.x86_64          libselinux-utils-3.6-1.el9.x86_64
#7 15.47   libutempter-1.2.1-6.el9.x86_64         openssl-1:3.2.2-2.el9.x86_64
#7 15.47   pam-1.5.1-20.el9.x86_64                policycoreutils-3.6-2.1.el9.x86_64
#7 15.47   systemd-252-37.el9.x86_64              systemd-libs-252-37.el9.x86_64
#7 15.47   systemd-pam-252-37.el9.x86_64          systemd-rpm-macros-252-37.el9.noarch
#7 15.47   unit-1.32.1-1.el9.ngx.x86_64           util-linux-2.37.4-18.el9.x86_64
#7 15.47   util-linux-core-2.37.4-18.el9.x86_64
#7 15.47
#7 15.47 Complete!
#7 DONE 15.6s

#8 exporting to image
#8 exporting layers
#8 exporting layers 0.7s done
#8 writing image sha256:5ca24a0b6321275864cf26fa0146fbbf20dada34282fa6afb23a915a24afc047 done
#8 DONE 0.8s

[ivanov17]

@thresheek you are right. It really works with dnf package manager. I have now tested this with the full base image.

Sorry, I forgot to mention that I'm using a minimal image. The minimal base image uses microdnf, which is the predecessor to dnf5. It looks like microdnf uses its own implementation to work with gpg keys. In both cases it returns the same error:

error: package unit-1.32.1-1.el9.ngx.x86_64 cannot be verified and repo unit is GPG enabled: /var/cache/yum/metadata/unit-9-x86_64/packages/unit-1.32.1-1.el9.ngx.x86_64.rpm could not be verified.
/var/cache/yum/metadata/unit-9-x86_64/packages/unit-1.32.1-1.el9.ngx.x86_64.rpm:  digest:  SIGNATURE:  NOT OK

However, there is some good news. I tried using the minimal Fedora 39 image. This image uses dnf5 as the package manager. And dnf5 correctly checks the signatures of both the repository and packages!

So, this problem only applies to microdnf. The bad news is that Red Hat UBI 9 images also use microdnf. And it looks like it will be used until 2032.

[thresheek]

microdnf definitely has its own set of bugs, that's for sure. This is really similar: rpm-software-management/libdnf#1320

I don't think we can do anything about it. This needs to be fixed in dnf/microdnf.

[ivanov17]

In any case, thanks for your help. I'm thinking about reporting this bug to Red Hat, but for now I'll just disable gpg checks. So, we can close this issue I guess.