Implement client side/boot time layering of extensions.

[jmarrero]

#300 took take for the yaml sugar and the build time treefile creation. However there is a second path to explore which layering package at boot time instead of just during the build. The idea is that someone would be able to use the same butane file to generate buildtime or boot time ignition files. The difference would be that the boot time we would create a service that runs rpm-ostree install PACKAGELIST instead of creating a tree file.

[jlebon]

Hmm, isn't this coreos/fedora-coreos-tracker#681? From the Butane side, there isn't anything to change.

I think you're arguing for baking coreos/fedora-coreos-tracker#681 (comment) into the OS (but picking up the new treefile dropin instead), which I'm OK with short-term. But long-term I think we want cleaner semantics as mentioned there.

[jmarrero]

I think it's similar to coreos/fedora-coreos-tracker#681 (comment) but ignoring the tree file dropin completely.

You would have a Dockerfile that looks like:
https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md#butane-as-a-declarative-input-format-for-layering

OR if it makes sense to use your liveapply work something like:

FROM quay.io/coreos-assembler/fcos
# Call butane:
butane  --pretty --strict --layer extentions.bu -o extentions.ign
# Ignition live-apply
ignition --ignition-liveapply extentions.ign

instead of calling rpm-ostree directly with a list of packages. We would make use of your work of ignition and result in a more declarative interface.

The new flag on butane (or it could check for a environment variable?) will then tell butane create systemd service instead of a tree file dropin.

[bgilbert]

There's a fairly high bar to adding command-line flags that change the semantics of Butane's output. We currently only have two, --files-dir and --raw; the former is needed to enforce a security boundary, and the latter produces (within the limits of each format) a semantically equivalent config in an alternate format. In general, the consequences of a config should be clear from its text, without knowing anything about the context in which it compiles.

In addition, we try not to embed significant OS code in Butane. It's possible to use old OS versions with newer Butane and vice versa, so any OS code that may need to evolve over time needs to be in the OS.

Can't we just have the OS read the treefile dropins on first boot and handle installing the packages?

[jlebon]

To re-iterate some of things mentioned in #304 and coreos/rpm-ostree#3340, the idea with the client-side treefile stuff is that it is both the interface for CoreOS layering and for rpm-ostree on the node (CoreOS layering provided the impetus for doing this, but it's been discussed well before in e.g. coreos/rpm-ostree#2326). So Butane itself should not need to change output. It's only how rpm-ostree applies the dropins that need to change (but there's some more design work needed there I think -- see the rebuild.cxx code in coreos/rpm-ostree#3340).

[jlebon]

If we want the new extensions: sugar to work today on first-boot, then what we can do is add something like coreos/fedora-coreos-tracker#681 (comment) in https://github.com/coreos/fedora-coreos-config, but instead of calling out to rpm-ostree install, it would call out to ex rebuild and then we fill in just enough of https://github.com/coreos/rpm-ostree/blob/c11b4fe23a848425b3ab6417a7185ec61b993b47/src/app/rpmostree-builtin-rebuild.cxx#L82-L83 to support the package layering bits. WDYT?

Edit: We'd want to be sure to signal somehow that this is experimental though and that the semantics will likely change in the future.

[cgwalters]

I think I would agree that ideally the goal here is achieved in both cases by using declarative bits in rpm-ostree, so all one needs out of butane/ignition is writing a file.

It's perhaps a bit like how we ended up adding the update ca trust service so that one didn't need to be embedded via butane or Ignition for this very common use case.

That said I also think there's some things that probably need configuration about exactly when rpm-ostree runs as part of the boot. This is a bit related to some nontrivial bits around network-online.target mentioned in https://docs.fedoraproject.org/en-US/fedora-coreos/os-extensions/ - xref coreos/rpm-ostree#3096

I guess if the systemd unit is shipped upstream in rpm-ostree, then all its "API" concerns move there and outside of butane.

[bgilbert]

If there are configuration knobs tightly coupled to extension support, it could make sense to have Butane sugar for them that transpiles to a config file.

[jmarrero]

OK, closing this in favor of doing: #322 (comment)