install: Add ensure-completion verb, wire up ostree-deploy → bootc

When bootc was created, it started to become a superset of ostree;
in particular things like `/usr/lib/bootc/kargs.d` and logically
bound images.

However...Anaconda today is still invoking `ostree container image deploy`.

Main fix
--------

When bootc takes over the `/usr/libexec/ostree/ext/ostree-container`
entrypoint, make the existing `ostree container image deploy` CLI actually
just call back into bootc to fix things up. No additional work required other
than getting an updated bootc in the Anaconda ISO.

Old Anaconda ISOs
-----------------

But, a further problem here is that Anaconda is only updated once
per OS major+minor - e.g. there won't be an update to it for the lifetime
of RHEL 9.5 or Fedora 41. We want the ability to ship new
features and bugfixes in those OSes (especially RHEL9.5).

So given that we have a newer bootc in the target container, we can
do this:

```
%post --erroronfail
bootc install ensure-completion
%end
```

And will fix things up. Of course there's fun $details here...the
way Anaconda implements `%post` is via a hand-augmented `chroot`
i.e. a degenerate container, and we need to escape that and
fix some things up (such as a missing cgroupfs mount).

Summmary
--------

- With a newer bootc in the ISO, everything just works
- For older ISOs, one can add the `%post` above as a workaround.

Implementation details: Cross-linking bootc and ostree-rs-ext
-------------------------------------------------------------

This whole thing is very confusing because now, the linkage
between bootc and ostree-rs-ext is bidirectional. In the case
of `bootc install to-filesystem`, we end up calling into ostree-rs-ext,
and we *must not* recurse back into bootc, because at least for
kernel arguments we might end up applying them *twice*. We do
this by passing a CLI argument.

The second problem is the crate-level dependency; right now they're
independent crates so we can't have ostree-rs-ext actually
call into bootc directly, as convenient as that would be. So we
end up forking ourselves as a subprocess. But that's not too bad
because we need to carry a subprocess-based entrypoint *anyways*
for the Anaconda `%post` case.

Implementation details: /etc/resolv.conf
----------------------------------------

There's some surprising stuff going on in how Anaconda handles
`/etc/resolv.conf` in the target root that I got burned by. In
Fedora it's trying to query if systemd-resolved is enabled in
the target or something?

I ended up writing some code to just try to paper over this
to ensure we have networking in the `%post` where we need
it to fetch LBIs.

Signed-off-by: Colin Walters <[email protected]>

lib/src/boundimage.rs

@@ -5,8 +5,6 @@
 //! pre-pulled (and in the future, pinned) before a new image root
 //! is considered ready.
 
-use std::num::NonZeroUsize;
-
 use anyhow::{Context, Result};
 use camino::Utf8Path;
 use cap_std_ext::cap_std::fs::Dir;
@@ -49,7 +47,7 @@ pub(crate) async fn pull_bound_images(sysroot: &Storage, deployment: &Deployment
 
 #[context("Querying bound images")]
 pub(crate) fn query_bound_images_for_deployment(
-    sysroot: &Storage,
+    sysroot: &ostree_ext::ostree::Sysroot,
     deployment: &Deployment,
 ) -> Result<Vec<BoundImage>> {
     let deployment_root = &crate::utils::deployment_fd(sysroot, deployment)?;
@@ -153,15 +151,21 @@ pub(crate) async fn pull_images(
     sysroot: &Storage,
     bound_images: Vec<crate::boundimage::BoundImage>,
 ) -> Result<()> {
-    tracing::debug!("Pulling bound images: {}", bound_images.len());
-    // Yes, the usage of NonZeroUsize here is...maybe odd looking, but I find
-    // it an elegant way to divide (empty vector, non empty vector) since
-    // we want to print the length too below.
-    let Some(n) = NonZeroUsize::new(bound_images.len()) else {
-        return Ok(());
-    };
     // Only do work like initializing the image storage if we have images to pull.
+    if bound_images.is_empty() {
+        return Ok(());
+    }
     let imgstore = sysroot.get_ensure_imgstore()?;
+    pull_images_impl(imgstore, bound_images).await
+}
+
+#[context("Pulling bound images")]
+pub(crate) async fn pull_images_impl(
+    imgstore: &crate::imgstorage::Storage,
+    bound_images: Vec<crate::boundimage::BoundImage>,
+) -> Result<()> {
+    let n = bound_images.len();
+    tracing::debug!("Pulling bound images: {n}");
     // TODO: do this in parallel
     for bound_image in bound_images {
         let image = &bound_image.image;

lib/src/cli.rs

@@ -182,6 +182,15 @@ pub(crate) enum InstallOpts {
     /// will be wiped, but the content of the existing root will otherwise be retained, and will
     /// need to be cleaned up if desired when rebooted into the new root.
     ToExistingRoot(crate::install::InstallToExistingRootOpts),
+    /// Intended for use in environments that are performing an ostree-based installation, not bootc.
+    ///
+    /// In this scenario the installation may be missing bootc specific features such as
+    /// kernel arguments, logically bound images and more. This command can be used to attempt
+    /// to reconcile. At the current time, the only tested environment is Anaconda using `ostreecontainer`
+    /// and it is recommended to avoid usage outside of that environment. Instead, ensure your
+    /// code is using `bootc install to-filesystem` from the start.
+    #[clap(hide = true)]
+    EnsureCompletion {},
     /// Output JSON to stdout that contains the merged installation configuration
     /// as it may be relevant to calling processes using `install to-filesystem`
     /// that in particular want to discover the desired root filesystem type from the container image.
@@ -346,6 +355,14 @@ pub(crate) enum InternalsOpts {
         #[clap(allow_hyphen_values = true)]
         args: Vec<OsString>,
     },
+    /// Invoked from ostree-ext to complete an installation.
+    BootcInstallCompletion {
+        /// Path to the sysroot
+        sysroot: Utf8PathBuf,
+
+        // The stateroot
+        stateroot: String,
+    },
 }
 
 #[derive(Debug, clap::Subcommand, PartialEq, Eq)]
@@ -989,6 +1006,10 @@ async fn run_from_opt(opt: Opt) -> Result<()> {
                 crate::install::install_to_existing_root(opts).await
             }
             InstallOpts::PrintConfiguration => crate::install::print_configuration(),
+            InstallOpts::EnsureCompletion {} => {
+                let rootfs = &Dir::open_ambient_dir("/", cap_std::ambient_authority())?;
+                crate::install::completion::run_from_anaconda(rootfs).await
+            }
         },
         #[cfg(feature = "install")]
         Opt::ExecInHostMountNamespace { args } => {
@@ -1026,6 +1047,10 @@ async fn run_from_opt(opt: Opt) -> Result<()> {
                 let sysroot = get_storage().await?;
                 crate::deploy::cleanup(&sysroot).await
             }
+            InternalsOpts::BootcInstallCompletion { sysroot, stateroot } => {
+                let rootfs = &Dir::open_ambient_dir("/", cap_std::ambient_authority())?;
+                crate::install::completion::run_from_ostree(rootfs, &sysroot, &stateroot).await
+            }
         },
         #[cfg(feature = "docgen")]
         Opt::Man(manopts) => crate::docgen::generate_manpages(&manopts.directory),

lib/src/install.rs

@@ -7,6 +7,7 @@
 // This sub-module is the "basic" installer that handles creating basic block device
 // and filesystem setup.
 pub(crate) mod baseline;
+pub(crate) mod completion;
 pub(crate) mod config;
 mod osbuild;
 pub(crate) mod osconfig;
@@ -762,6 +763,7 @@ async fn install_container(
     )?;
     let kargsd = kargsd.iter().map(|s| s.as_str());
 
+    // Keep this in sync with install/completion.rs for the Anaconda fixups
     let install_config_kargs = state
         .install_config
         .as_ref()
@@ -786,6 +788,7 @@ async fn install_container(
     options.kargs = Some(kargs.as_slice());
     options.target_imgref = Some(&state.target_imgref);
     options.proxy_cfg = proxy_cfg;
+    options.skip_completion = true; // Must be set to avoid recursion!
     options.no_clean = has_ostree;
     let imgstate = crate::utils::async_task_with_spinner(
         "Deploying container image",
@@ -1383,7 +1386,7 @@ async fn install_with_sysroot(
             }
         }
         BoundImages::Unresolved(bound_images) => {
-            crate::boundimage::pull_images(sysroot, bound_images)
+            crate::boundimage::pull_images_impl(imgstore, bound_images)
                 .await
                 .context("pulling bound images")?;
         }