dnf5 update fails with gpgcheck=1

[ctubbsii]

Service(s)

pkg.jenkins.io

Summary

I recently upgraded to Fedora 41, which now uses dnf5. dnf5 works fine for all my other configured yum repos, but for pkg.jenkins.io/redhat, a dnf update fails with the following error message:

Transaction failed: Signature verification failed.
PGP check for package "jenkins-2.487-1.1.noarch" (/var/cache/libdnf5/jenkins-3de128c1ecbdf1be/packages/jenkins-2.487-1.1.noarch.rpm) from repo "jenkins" has failed: The repository does not have any PGP keys configured.

The only workaround that I know works is setting gpgcheck=0:

[jenkins]
name=Jenkins
baseurl=https://pkg.jenkins.io/redhat
gpgcheck=0 # should be gpgcheck=1, but that doesn't work with dnf5 for some reason

Reproduction steps

1. Follow the instructions at https://pkg.jenkins.io/redhat/ to install jenkins
2. Execute dnf update when an update is available

[MarkEWaite]

The Fedora instructions in the Jenkins handbook work with Fedora 41. The instructions at pkg.jenkins.io are trying to be more general purpose so that they run on all Red Hat distributions. We have a pending issue that proposes to replace those instructions with a link to the installation page of the Jenkins handbook.

[kmartens27]

I have created jenkinsci/packaging#495 to try and address this issue, as it will require review from the infra-team and other maintainers.
I have also created jenkins-infra/jenkins.io#7726 so that we do not lose the historical information about which Java version is supported

[MarkEWaite]

Next weekly build will update https://pkg.jenkins.io/redhat/ with the updated content. Thanks @kmartens27 !

[kmartens27]

Thanks very much @MarkEWaite

[ctubbsii]

The instructions at pkg.jenkins.io are trying to be more general purpose so that they run on all Red Hat distributions. We have a pending issue that proposes to replace those instructions with a link to the installation page of the Jenkins handbook.

The problem I'm reporting is not a documentation problem, and has nothing to do with which instructions are followed. The only important difference between the two sets of instructions are whether the packages come from the /redhat repo (which is more or less bleeding edge) or the /redhat-stable repo (which is a few versions behind, presumably to undergo additional testing). However, BOTH sets of instructions result in the same "The repository does not have any PGP keys configured" error.

As it turns out, the problem was that the Fedora upgrade instructions suggest cleaning up old keys that are no longer referenced in /etc/yum.repos.d with the command sudo clean-rpm-gpg-pubkey. Since jenkins' installation instructions involve manually installing the key, and the jenkins.repo file doesn't actually reference the key, jenkin's keys get deleted from the rpm database during this step, and needs to be reinstalled.

As an aside, the jenkins.repo files (in both sets of instructions) use http instead of https for the repository, which makes the situation worse, if users resort to disabling gpg checks using gpgcheck=0, because then there's no security at all on acquiring the RPMs from the yum repository. It'd be better if the jenkins.repo file used an https URL.

I propose two updates to the jenkins.repo files to improve the situation:

1. Add the line: gpgkey=https://pkg.jenkins.io/redhat/jenkins.io-2023.key
2. Use https instead of http for the baseurl

As an alternative to number 1, the jenkins RPM should be changed to install the jenkins.repo file and to install a copy of the key locally to /etc/pki/rpm-gpg/RPM-GPG-KEY-jenkins.io.2023 and the line in the jenkins.repo file should be: gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-jenkins.io.2023. Then, the instructions should be updated to install the key (same as current instructions) then to install the current RPM instead of the repo (this relies on the RPM to install the repo for updates). This suggestion is similar to what google-chrome does (the RPM installs the repo), and would simplify the installation steps to two: 1. install key, 2. install RPM, instead of the three steps currently: 1. install key, 2. install repo, 3. install RPM.

[MarkEWaite]

Thanks for the additional insights @ctubbsii ! Much appreciated!

As it turns out, the problem was that the Fedora upgrade instructions suggest cleaning up old keys that are no longer referenced in /etc/yum.repos.d with the command sudo clean-rpm-gpg-pubkey. Since jenkins' installation instructions involve manually installing the key, and the jenkins.repo file doesn't actually reference the key, jenkin's keys get deleted from the rpm database during this step, and needs to be reinstalled.

Doesn't that mean that you didn't actually follow the instructions on the pkg.jenkins.io page? Those instructions include the step:

 sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io-2023.key

I thought that would install the Jenkins GPG key until the next time that someone runs sudo clean-rpm-gpg-pubkey. Have I misunderstood the way that clean-rpm-gpg-pubkey works?

My test installation was using a freshly installed Fedora 41 machine, without any upgrade. I followed the instructions from the installation guide with no issue. Based on your description of the root cause of the issue, I suspect that the pkg.jenkins.io instructions would have worked for me as well.

I propose two updates to the jenkins.repo files to improve the situation:

1. Add the line: gpgkey=https://pkg.jenkins.io/redhat/jenkins.io-2023.key
2. Use https instead of http for the baseurl

I like those proposals very much! Will they work as expected with RHEL 8 (and derivatives) and RHEL 9 (and derivatives)?

As an alternative to number 1, the jenkins RPM should be changed to install the jenkins.repo file and to install a copy of the key locally to /etc/pki/rpm-gpg/RPM-GPG-KEY-jenkins.io.2023 and the line in the jenkins.repo file should be: gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-jenkins.io.2023. Then, the instructions should be updated to install the key (same as current instructions) then to install the current RPM instead of the repo (this relies on the RPM to install the repo for updates).

That sounds very attractive as well. Does that work with RHEL 8 (and derivatives) and RHEL 9 (and derivatives)?

[ctubbsii]

Doesn't that mean that you didn't actually follow the instructions on the pkg.jenkins.io page?

No. Perhaps I wasn't clear, but I encountered this after upgrading Fedora to 41. I followed those instructions to install Jenkins prior to upgrading Fedora. I did not expect to have to revisit any part of the Jenkins installation instructions after the Fedora 41 upgrade, since Jenkins was already installed.

I thought that would install the Jenkins GPG key until the next time that someone runs sudo clean-rpm-gpg-pubkey.

You are correct. It was just unexpected (to me) that the jenkins key would have been removed, since it was still in use (Jenkins was still installed and still required updates), and I didn't notice that jenkins' key was included in the long list of old keys it removed. I found that it was because there was no gpgkey= directive in the jenkins.repo to indicate the key was still in use (I also confirmed that a proper entry for this will prevent clean-rpm-gpg-pubkey from removing Jenkins' key).

I like those proposals very much! Will they work as expected with RHEL 8 (and derivatives) and RHEL 9 (and derivatives)?

Yes, definitely they will. I was able to confirm with multiple 3rd-party repos on a RHEL8 machine that both suggestions will work on RHEL8 and RHEL9 (docker-ce, epel, and redhat-rhui all use https for baseurl= and mirrorlist= entries, and also use gpgkey=file:///path/to/local/key).

As an alternative to number 1...

That sounds very attractive as well. Does that work with RHEL 8 (and derivatives) and RHEL 9 (and derivatives)?

Yes, there are no additional features needed for this alternative that differ from the previous suggestion. It just has a few advantages, like:

1. able to resolve the key locally instead of re-retrieval from jenkins.io,
2. able to manage updates to the jenkins.repo file with the RPM, and
3. reducing the number of steps for the initial installation

[dduportal]

Reopening the issue as it looks like there might be some work to be done to improve the situation.

@MarkEWaite I propose we'll discuss this during the 10 Dec. Infra meeting