Implement system-upgrade and arbitrary offline transactions #1280

evan-goode · 2024-02-27T20:36:15Z

This PR supersedes #1209.

Resolves #1052.

Together with #1264, resolves #1224.

This patch implements dnf5 system-upgrade as a regular command rather than a plugin. system-upgrade is responsible for performing offline upgrades of the system to a new major release. It has three main subcommands:

dnf5 system-upgrade download --releasever XX: Does a dry run of an upgrade to a new releasever and stores the resulting transaction for later.
dnf5 system-upgrade reboot: Creates a "magic symlink" at /system-update to trigger the start of dnf5-offline-transaction.service on the next boot, then reboots. See https://www.freedesktop.org/software/systemd/man/latest/systemd.offline-updates.html. dnf5-offline-transaction.service will take the place of the usual boot target so we can preform the upgrade without disturbing too many running processes.
dnf5 system-upgrade _execute: Not intended to be run by the user directly, but rather by dnf5-offline-transaction.service. Loads the transaction saved by system-upgrade download and performs the upgrade. Interacts with Plymouth (if it's available) to show upgrade progress on the animated boot screen. The usual DNF 5 stdout is additionally logged to journald. Reboots or powers off when the upgrade is complete.

This PR also adds a new command, dnf5 offline, which shares the reboot, log, and clean subcommands with dnf5 system-upgrade, and adds a new --offline argument that allows any operation to be performed offline, as proposed here: #1224 (comment). For example, the process for running an offline upgrade would look like:

$ sudo dnf5 upgrade --offline mypackage
Updating and loading repositories:
Repositories loaded.
...
Transaction stored to be performed offline. Run `dnf5 offline reboot` to reboot and run the transaction.
$ sudo dnf5 offline reboot
The system will now reboot to perform the offline transaction initiated by the following command:
	dnf5 upgrade --offline mypackage
Is this ok [y/N]:

The following commands support --offline: autoremove, distro-sync, downgrade, install, group install, group upgrade, reinstall, remove, swap, and upgrade.

A caveat of making system-upgrade and offline regular commands instead of plugins is that libsystemd and sdbus-c++ now have to be build dependencies of dnf5, since dnf5 offline log uses these libraries to display logs from offline transactions.

Remaining tasks:

Add dnf5 offline-distrosync and dnf5 offline-upgrade aliases to dnf5 distro-sync --offline and dnf5 upgrade --offline, respectively
Add systemd build condition so we don't have to always depend on libsystemd+sdbus-c++
dnf5 offline status subcommand
Documentation
Go back through commit history of dnf-plugins-extras to make sure no historical issues with DNF 4 system-upgrade are present in the new implementation

I'll implement the tests in a separate PR if that works. We have the DNF 4 tests to go from: https://github.com/rpm-software-management/dnf-plugins-core/blob/master/tests/test_system_upgrade.py

kontura · 2024-02-28T10:43:35Z

I see this is still a draft but I would prefer if the commits got restructured.

Specifically I think adding the plugin and then removing it right away is just complicating things, I would squash the commits.

Also I think the system-upgrade: Refactor, allow any offline transaction commit could be split it smaller ones.
Basically each bullet point could be a separate commit.

evan-goode · 2024-02-28T14:53:05Z

I see this is still a draft but I would prefer if the commits got restructured.

Specifically I think adding the plugin and then removing it right away is just complicating things, I would squash the commits.

Also I think the system-upgrade: Refactor, allow any offline transaction commit could be split it smaller ones. Basically each bullet point could be a separate commit.

Yeah, that's reasonable, I'll reorganize this. Since I'm doing it after the fact, it might not be super tidy (each intermediate commit may not compile), but the history will be more readable.

evan-goode · 2024-02-29T02:00:08Z

OK, I've squashed the commits from the old PR (#1209) and split the large commit into (1) the addition of the offline and system-upgrade commands and (2) the --offline flag. More squashing might be needed as I add more things, but reviews now are appreciated.

Conan-Kudo · 2024-02-29T02:26:47Z

This needs a rebase on current mainline. It does not apply.

dnf5/commands/offline/offline.cpp

dnf5/commands/system-upgrade/system-upgrade.hpp

include/libdnf5/offline/offline.hpp

kontura · 2024-03-04T10:12:59Z

include/libdnf5/offline/offline.hpp

+    void read();
+    std::exception_ptr read_exception;
+    std::filesystem::path path;
+    OfflineTransactionStateData data;


Could we hide this into a p_impl?
Perhaps also the OfflineTransactionStateData could store the data in p_impl and have a bunch of getters/setters for the values?

This would allow us to do modifications later without breaking ABI.

I think we no longer need to do this if the offline module is not in libdnf5, right?

kontura · 2024-03-04T10:15:37Z

include/libdnf5/offline/offline.hpp

+TOML11_DEFINE_CONVERSION_NON_INTRUSIVE(
+    libdnf5::offline::OfflineTransactionStateData,
+    state_version,
+    status,
+    cachedir,
+    target_releasever,
+    system_releasever,
+    verb,
+    cmd_line,
+    poweroff_after,
+    enabled_repos,
+    disabled_repos)


Could this also be moved to the cpp file?
The conversions are only used in offline.cpp right?

Yes, but IMHO it's best to keep it in the header since all users of the struct should agree on how to serialize/deserialize it.

Conan-Kudo · 2024-03-04T10:40:13Z

It probably makes sense to offer a build conditional for this since dnf5 on non-systemd distributions (such as Yocto) is going to be a thing.

dnf5/include/dnf5/context.hpp

include/libdnf5/offline/offline.hpp

dnf5/commands/system-upgrade/system-upgrade.cpp

evan-goode · 2024-03-12T04:37:58Z

It probably makes sense to offer a build conditional for this since dnf5 on non-systemd distributions (such as Yocto) is going to be a thing.

Done in 705de82, thanks for the reminder.

evan-goode · 2024-03-12T04:39:59Z

I'm marking this as RFR now, I think it's in a good state. I'd like to add the tests in a separate PR.

Implement `offline` and `system-upgrade` commands. `dnf5 system-upgrade` is intended to be backwards-compatible with `dnf4 system-upgrade`. All of system-upgrade's functionality, with the exception of the `download` subcommand, is implemented in the new `offline` command, and system-upgrade shares subcommands with it. This is more or less a port of `dnf4 system-upgrade`. This behavior of `dnf5 offline` was proposed here: rpm-software-management#1224 (comment). For rpm-software-management#1052

Adds a new --offline argument that allows any operation to be performed offline, as proposed here: rpm-software-management#1224 (comment). The following commands support `--offline`: `autoremove`, `distro-sync`, `downgrade`, `install`, `group install`, `group upgrade`, `reinstall`, `remove`, `swap`, and `upgrade`.

Addresses rpm-software-management/dnf-plugins-core#512

- Disable autocomplete for dnf5 offline _execute - Collapse SystemUpgradeSubcommand and its subclass SystemUpgradeDownloadCommand - Move include/libdnf5/offline/offline.hpp to dnf5/include/dnf5/offline.hpp - Add journald messages that use OFFLINE_FINISHED_ID, REBOOT_REQUESTED_ID, and DOWNLOAD_FINISHED_ID - Make Context.should_store_offline private with a getter and setter - Make cacheonly setting in `dnf5 offline _execute` actually work by applying it in `pre_configure`, not `configure` - Translate some strings

--installroot with `system-upgrade` and `offline` should now work as it does in DNF 4.

- Store module_platform_id in offline-transaction-state.toml - Set nogpgcheck in `dnf5 offline execute` since the gpgcheck should have already been performed when preparing the transaction

Document the system-upgrade and offline commands and the --offline flag.

dnf5/commands/offline/offline.cpp

dnf5.spec

dnf5/context.cpp

dnf5/commands/offline/offline.cpp

doc/commands/offline.8.rst

dnf5/CMakeLists.txt

dnf5.spec

kontura

It seems there is some issue with systemd, its not building on f38 and f39.

On Monday I would also like to run the CLI by everyone again to make sure they are ok with it. It is a lot of changes.

kontura · 2024-03-15T12:31:36Z

dnf5/config/usr/share/dnf5/aliases.d/compatibility.conf

+['offline-distrosync']
+type = 'command'
+attached_command = 'distro-sync'
+group_id = 'commands-compatibility-aliases'
+complete = true
+attached_named_args = [
+  { id_path = 'distro-sync.offline' }
+]
+
+['offline-upgrade']
+type = 'command'
+attached_command = 'upgrade'
+group_id = 'commands-compatibility-aliases'
+complete = true
+attached_named_args = [
+  { id_path = 'upgrade.offline' }
+]
+


I see these aliases don't match the dnf4 commands.
It basically changes dnf offline-distrosync download -> dnf5 offline-distrosync and discards the rest.

We do have all the functionality so I think we could build the aliases to (more or less) fit.

Something like:

['offline-distrosync'] type = 'command' attached_command = 'distro-sync' group_id = 'commands-compatibility-aliases' complete = true ['offline-distrosync.download'] type = 'command' attached_command = 'distro-sync' complete = true attached_named_args = [ { id_path = 'distro-sync.offline' } ] ['offline-distrosync.log'] type = 'command' attached_command = 'offline.log' complete = true ...

(If we do decide to do that I would rather do it in a separate PR.)

evan-goode · 2024-03-18T14:27:44Z

It seems there is some issue with systemd, its not building on f38 and f39.

Alright, I think it's fixed. We needed to get the SYSTEMD_SYSTEM_UNIT_DIR variable from a pkgconfig file provided by systemd, not libsystemd-devel.

kontura · 2024-03-19T06:23:06Z

Thank you!

evan-goode mentioned this pull request Feb 27, 2024

Implement system-upgrade plugin #1209

Closed

5 tasks

evan-goode force-pushed the evan-goode/offline-transactions branch from 3b12b42 to 589a4e6 Compare February 27, 2024 21:15

jan-kolarik self-requested a review February 28, 2024 07:24

evan-goode force-pushed the evan-goode/offline-transactions branch from 589a4e6 to 94c3d97 Compare February 29, 2024 01:57

kontura reviewed Mar 4, 2024

View reviewed changes

jan-kolarik reviewed Mar 4, 2024

View reviewed changes

dnf5/include/dnf5/context.hpp Outdated Show resolved Hide resolved

jan-kolarik reviewed Mar 4, 2024

View reviewed changes

include/libdnf5/offline/offline.hpp Outdated Show resolved Hide resolved

jan-kolarik reviewed Mar 4, 2024

View reviewed changes

dnf5/commands/system-upgrade/system-upgrade.cpp Outdated Show resolved Hide resolved

kontura self-assigned this Mar 4, 2024

evan-goode force-pushed the evan-goode/offline-transactions branch from 94c3d97 to 56c9a02 Compare March 4, 2024 20:35

jan-kolarik mentioned this pull request Mar 7, 2024

Implement history command #140

Closed

evan-goode force-pushed the evan-goode/offline-transactions branch 2 times, most recently from 8e39a69 to 995d379 Compare March 8, 2024 20:03

evan-goode marked this pull request as ready for review March 12, 2024 04:40

evan-goode added 9 commits March 12, 2024 04:40

system-upgrade: Add warning to dnf5 offline _execute

7d690a0

Addresses rpm-software-management/dnf-plugins-core#512

system-upgrade: Add OFFLINE_STARTED_ID comment, cleanup

9f3909d

system-upgrade: Handle _execute errors

a0886d2

system-upgrade: Fix installroot

e0a09b6

--installroot with `system-upgrade` and `offline` should now work as it does in DNF 4.

system-upgrade: offline-distrosync, offline-upgrade aliases

73a7da5

system-upgrade: fix DNF 4 regressions

20dab2a

- Store module_platform_id in offline-transaction-state.toml - Set nogpgcheck in `dnf5 offline execute` since the gpgcheck should have already been performed when preparing the transaction

evan-goode added 3 commits March 12, 2024 04:41

system-upgrade: offline status subcommand

3fb5fb8

system-upgrade: systemd behind build flag

56a4a61

system-upgrade: Documentation

520d21b

Document the system-upgrade and offline commands and the --offline flag.

evan-goode force-pushed the evan-goode/offline-transactions branch from bae9603 to 520d21b Compare March 12, 2024 04:41

evan-goode requested a review from kontura March 12, 2024 04:41

kontura reviewed Mar 14, 2024

View reviewed changes

dnf5/commands/offline/offline.cpp Outdated Show resolved Hide resolved

dnf5.spec Outdated Show resolved Hide resolved

dnf5/context.cpp Show resolved Hide resolved

dnf5/commands/offline/offline.cpp Show resolved Hide resolved

doc/commands/offline.8.rst Show resolved Hide resolved

Conan-Kudo requested changes Mar 14, 2024

View reviewed changes

dnf5/CMakeLists.txt Outdated Show resolved Hide resolved

dnf5/CMakeLists.txt Outdated Show resolved Hide resolved

dnf5.spec Outdated Show resolved Hide resolved

system-upgrade: misc fixes

377527c

evan-goode force-pushed the evan-goode/offline-transactions branch from 6799f77 to 377527c Compare March 15, 2024 05:08

evan-goode requested review from Conan-Kudo and kontura March 15, 2024 05:17

kontura reviewed Mar 15, 2024

View reviewed changes

system-upgrade: fix pkgconfig systemd variable

b402ddf

kontura added this pull request to the merge queue Mar 19, 2024

Merged via the queue into rpm-software-management:main with commit e8c5901 Mar 19, 2024
13 of 22 checks passed

kontura mentioned this pull request Mar 20, 2024

offline-distrosync compatible subcommands #1330

Closed

evan-goode mentioned this pull request Mar 25, 2024

system-upgrade: Add offline-distrosync, offline-upgrade subcommand aliases #1343

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement system-upgrade and arbitrary offline transactions #1280

Implement system-upgrade and arbitrary offline transactions #1280

evan-goode commented Feb 27, 2024 •

edited

Loading

kontura commented Feb 28, 2024

evan-goode commented Feb 28, 2024

evan-goode commented Feb 29, 2024

Conan-Kudo commented Feb 29, 2024

kontura Mar 4, 2024

evan-goode Mar 6, 2024

kontura Mar 4, 2024

evan-goode Mar 6, 2024

Conan-Kudo commented Mar 4, 2024

evan-goode commented Mar 12, 2024

evan-goode commented Mar 12, 2024

kontura left a comment

kontura Mar 15, 2024 •

edited

Loading

evan-goode commented Mar 18, 2024

kontura commented Mar 19, 2024

Implement system-upgrade and arbitrary offline transactions #1280

Implement system-upgrade and arbitrary offline transactions #1280

Conversation

evan-goode commented Feb 27, 2024 • edited Loading

kontura commented Feb 28, 2024

evan-goode commented Feb 28, 2024

evan-goode commented Feb 29, 2024

Conan-Kudo commented Feb 29, 2024

kontura Mar 4, 2024

Choose a reason for hiding this comment

evan-goode Mar 6, 2024

Choose a reason for hiding this comment

kontura Mar 4, 2024

Choose a reason for hiding this comment

evan-goode Mar 6, 2024

Choose a reason for hiding this comment

Conan-Kudo commented Mar 4, 2024

evan-goode commented Mar 12, 2024

evan-goode commented Mar 12, 2024

kontura left a comment

Choose a reason for hiding this comment

kontura Mar 15, 2024 • edited Loading

Choose a reason for hiding this comment

evan-goode commented Mar 18, 2024

kontura commented Mar 19, 2024

evan-goode commented Feb 27, 2024 •

edited

Loading

kontura Mar 15, 2024 •

edited

Loading