Proposal: Utilize sysext-kitchen for our systemd extensions

[Zeglius]

I propose adding https://github.com/Zeglius/sysext-kitchen as an organization repo, and use such as the core for building and shipping our Systemd extensions.

Currently it is not production-ready, and needs further development (at the time of writing, I'm working in a refactor that allows us to build multiple extensions in one go, see multi-image branch), but that's the main reason why I want to submit this to the org, as it would help to reduce the workload off my shoulders.

And yeah, I noticed there is https://github.com/ublue-os/sysext, based in flatcar, and if I were to list the pros of using sysext-kitchen over flatcar, these would be:

1. Sysext-kitchen is 99% powered by mkosi, which is a project under the direct umbrella of Systemd org, and thus is more tightly coupled with Systemd extensions development on the long run.

2. Mkosi has a more consistent and restrictive yet flexible systemd extension building configuration system compared to flatcar, utilizing a well structured layout of conf files and drop-ins, compared to flatcar, that is composed mostly of scripts with a less consistent/restrictive layout.

3. Mkosi has a really consistent and concise image building system, see Execution Flow, which would allow to avoid future complexity and inconsistency, as flatcar seems way too less restrictive.

4. Mkosi is not limited to systemd extensions, it can as well deal with signing sysext using MOK(?).
   I'm not gonna delve further in this point, I feel there is more experienced that should take deeper look into this, It would be interesting to use this system to reinforce security in our sysexts.

"We can’t sign the extension image without a key, so let’s generate one with mkosi genkey (or write your own private key and certificate yourself to mkosi.key and mkosi.crt respectively). Note that this key will need to be loaded into your kernel keyring either at build time or via MOK for systemd to accept the system extension at runtime as trusted."

https://mkosi.systemd.io/sysext.html#:~:text=We%20can%E2%80%99t%20sign,runtime%20as%20trusted.

There is as well fedora-sysext, but suffers of the same issues than flatcar.

[travier]

I'm using two "modes" to build the sysexts in https://github.com/travier/fedora-sysexts:

• The first one via justfiles which are indeed just bash scripts with some structure. It leverages systemd-sysupdate for updates but there are no guarantee that the sysexts will thus match the version of the OS you have installed as it will always update to the latest. So this works but is not ideal and probablky what we should use longer term.
• The other one via Containerfiles which I think should probably be the future as it will make integrating with bootc easier: sysexts will be "just another tag" to pull as part of an update, similarly to logically bound containers: see https://github.com/travier/fedora-sysexts#how-does-this-fit-with-bootc-and-bootable-containers.

But overall, this is a great initiative.

Some questions:

• How does the mkosi approach works to install just what's needed?
• What the output format?
• How would the sysexts be delivered and keept up to date in sync with the system?

[Zeglius]

How does the mkosi approach works to install just what's needed

It can be feed with root file systems and use them as a "base". Right now, I'm fetching a container image from ours (per example, bazzite) and extract that very root filesystem.

After that, it mounts an overlayfs and installs the requested packages/files, and later on takes that overlay upper layer plus other necessary files like /usr/lib/os-release and outputs it as an "image" (can be a directory, a raw partition file etc)

Then, in an additional step, mkosi takes that output and feeds into both a sysext and confext "step". In these final touches are made, like preparing extension-release.NAME files, etc, and the final phase it uses mkfs.erofs to create the final raw file.

TL;DR: Basically it uses overlayfs to do diffing of the base/sysext files plus some other files.

What the output format?

Right now, erofs raw image files, though an additional step can be added to extract these into directories or tars if wished.

How would the sysexts be delivered and keept up to date in sync with the system?

Ideally with containers/bootc#7 or rpm-software-management/dnf5#1731, but in early stages as of right now, Im thinking of sysupdate with some tweaks like using the ostree lock file with a systemd.path unit to sync both updates

[Zeglius]

Also forgot to mention, mkosi allows for OCI images as output, so in theory should be easy to plug-in with a container based ecosystem

[Zeglius]

Okay so mkosi can't take a sysext as input and output as an OCI, due some incompatibility with the file tree of a sysext.

So ended up uploading it as a raw file inside an OCI artifact using oras.

Far from ideal, would require to create additional tooling, and we still don't even know how systemd extensions will be supposed to work when it comes to building them.

My assumption is something similar to what Timothée already does with fedora-sysext with containers (?).

So I'm gonna close this issue. It works(?) (for me) but not sure if this is what we should push for building our sysexts.