Bootloader Plugin: efivar

[Promaethius]

Are there any plans to implement efivar as a bootloader?
https://github.com/rhboot/efivar
For example, compiling the kernel with efi stubs and using secure boot direct to kernel.

[bauen1]

There's a number of different issues asking for different bootloader setup support: e.g.

• systemd-boot support #1719
• Support UKI #2753
• sysroot: Support customising /boot on root for syslinux and u-boot #2145
• bootloader: Add an aboot (Android) bootloader backend #2793

Ideally libostree only has to generate a BLS configuration and be done with it, however there are setups that would require special handling to integrate nicely with ostree admin deploy.

Would it be possible to provide a "bootloader" that simply runs hook scripts from the deployment ?
While far from ideal, it would make prototyping a lot easier, and less dirty than a fake grub-mkconfig script, like I'm doing for my UKI+SecureBoot setup.
And it would allow for exotic setups that have little to no chance of ever being upstreamed.
Ideally without chrooting, since that is complicated and if necessary could be done in those scripts.

[dbnicholson]

That's an interesting idea. Another way to potentially do it is to make the bootloader managers separate executables that are installed in /usr/lib/ostree/bootloader or similar. Then change the loading logic to match by basename. Then you could just drop in /usr/lib/ostree/bootloader/mycustomthing and then ostree config set sysroot.bootloader mycustomthing on the host.

Also, for your scenario I'd suggest explicitly setting the bootloader to none with ostree config set sysroot.bootloader none. Then all it does is create the boot loader config entries and you can do any hacks you need after deploy. We do this on Endless because we have grub patched to support BLS and do not want anything fiddling with grub.cfg.

[bauen1]

That's an interesting idea. Another way to potentially do it is to make the bootloader managers separate executables that are installed in /usr/lib/ostree/bootloader or similar. Then change the loading logic to match by basename. Then you could just drop in /usr/lib/ostree/bootloader/mycustomthing and then ostree config set sysroot.bootloader mycustomthing on the host.

Wouldn't separate executables make things more difficult than necessary ?
And aboot seems to require special handling in ostree-sysroot-deploy.c.

While writing a custom bootloader manager that simply calls my scripts seems to be very simple, perhaps a 100 lines of code at most, I sadly don't have the time currently to setup an environment to implement and test it.

Just a couple more thoughts:

1. The script to call should probably be configurable e.g. by settting a sysroot.bootloader_script config on the repository.
2. The sysroot to use should be passed to the script, since there seems to be no other way of obtaining it, and it is necessary to make e.g. installers work nicely.
3. The bootversion needs to be passed along, it could be figured out by the script from looking at the boot/loader link, but the information is already easily available
4. Passing the new_deployments to the script seems unnecessary, since the kernel arguments, the kernel and initrd path can be obtained from parsing the BLS config files under boot/loader.$bootversion/entries/

I can't think of any scenario where chrooting before calling the script would be desirable.
Maybe when run from an installer, but I think implementing chroot in the script itself would be easier and more flexible.

In general I've noticed that there are quite a few different ways ostree-based chroots are setup, of course they are all slightly different.
For my setup alone I can count:

1. The installer setting up the installed deployment before the first boot
2. The initramfs (Granted, I'm currently using some shell script and not ostrees binary)
3. ostree-bootloader-grub2.c:

   ostree/src/libostree/ostree-bootloader-grub2.c

   Lines 275 to 328 in 581a580

   	/* Post-fork, pre-exec child setup for grub2-mkconfig */
   	static void
   	grub2_child_setup (gpointer user_data)
   	{
   	Grub2ChildSetupData *cdata = user_data;
   	setenv ("_OSTREE_GRUB2_BOOTVERSION", cdata->bootversion_str, TRUE);
   	/* We have to pass our state (whether or not we're using EFI) to the child */
   	if (cdata->is_efi)
   	setenv ("_OSTREE_GRUB2_IS_EFI", "1", TRUE);
   	/* Everything below this is dealing with the chroot case; if
   	* we're not doing that, return early.
   	*/
   	if (!cdata->root)
   	return;
   	/* TODO: investigate replacing this with bwrap */
   	if (chdir (cdata->root) != 0)
   	{
   	perror ("chdir");
   	_exit (1);
   	}
   	if (unshare (CLONE_NEWNS) != 0)
   	{
   	perror ("CLONE_NEWNS");
   	_exit (1);
   	}
   	if (mount (NULL, "/", "none", MS_REC|MS_PRIVATE, NULL) < 0)
   	{
   	perror ("Failed to make / a private mount");
   	_exit (1);
   	}
   	if (mount (".", ".", NULL, MS_BIND | MS_PRIVATE, NULL) < 0)
   	{
   	perror ("mount (MS_BIND)");
   	_exit (1);
   	}
   	if (mount (cdata->root, "/", NULL, MS_MOVE, NULL) < 0)
   	{
   	perror ("failed to MS_MOVE to /");
   	_exit (1);
   	}
   	if (chroot (".") != 0)
   	{
   	perror ("chroot");
   	_exit (1);
   	}
   	}

   ; Except that is only used when not booted into ostree
4. Running semodule --refresh

   ostree/src/libostree/ostree-sysroot-deploy.c

   Lines 2911 to 2964 in d49632f

   	/*
   	* Derived from rpm-ostree's rust/src/bwrap.rs
   	*/
   	static gboolean
   	run_in_deployment (int deployment_dfd,
   	const gchar * const *child_argv,
   	gsize child_argc,
   	gint *exit_status,
   	gchar **stdout,
   	GError **error)
   	{
   	static const gchar * const COMMON_ARGV[] = {
   	"/usr/bin/bwrap",
   	"--dev", "/dev", "--proc", "/proc", "--dir", "/run", "--dir", "/tmp",
   	"--chdir", "/",
   	"--die-with-parent",
   	"--unshare-pid",
   	"--unshare-uts",
   	"--unshare-ipc",
   	"--unshare-cgroup-try",
   	"--ro-bind", "/sys/block", "/sys/block",
   	"--ro-bind", "/sys/bus", "/sys/bus",
   	"--ro-bind", "/sys/class", "/sys/class",
   	"--ro-bind", "/sys/dev", "/sys/dev",
   	"--ro-bind", "/sys/devices", "/sys/devices",
   	"--bind", "usr", "/usr",
   	"--bind", "etc", "/etc",
   	"--bind", "var", "/var",
   	"--symlink", "/usr/lib", "/lib",
   	"--symlink", "/usr/lib32", "/lib32",
   	"--symlink", "/usr/lib64", "/lib64",
   	"--symlink", "/usr/bin", "/bin",
   	"--symlink", "/usr/sbin", "/sbin",
   	};
   	static const gsize COMMON_ARGC = sizeof (COMMON_ARGV) / sizeof (*COMMON_ARGV);
   	gsize i;
   	GPtrArray *args = g_ptr_array_sized_new (COMMON_ARGC + child_argc + 1);
   	g_autofree gchar **args_raw = NULL;
   	for (i = 0; i < COMMON_ARGC; i++)
   	g_ptr_array_add (args, (gchar *) COMMON_ARGV[i]);
   	for (i = 0; i < child_argc; i++)
   	g_ptr_array_add (args, (gchar *) child_argv[i]);
   	g_ptr_array_add (args, NULL);
   	args_raw = (gchar **) g_ptr_array_free (args, FALSE);
   	return g_spawn_sync (NULL, args_raw, NULL, 0, &child_setup_fchdir,
   	(gpointer) (uintptr_t) deployment_dfd,
   	stdout, NULL, exit_status, error);
   	}

Also, for your scenario I'd suggest explicitly setting the bootloader to none with ostree config set sysroot.bootloader none. Then all it does is create the boot loader config entries and you can do any hacks you need after deploy. We do this on Endless because we have grub patched to support BLS and do not want anything fiddling with grub.cfg.

Yes, I could, however I want to have ostree admin deploy work seamlessly.
My hack is already quite close to libostree just calling a script, ideally I will be able to replace it with e.g. #2753 or libostree calling my scripts.

I've also found that I need to "wrap" grub-mkconfig in my modified Debian Installer (which is an even bigger hack), since the chroot provided by ostree-bootloader-grub2.c is lacking every important filesystem, not sure if that's because I'm doing something wrong, but it seems that

ostree/src/libostree/ostree-bootloader-grub2.c

Lines 275 to 328 in 490f515

	/* Post-fork, pre-exec child setup for grub2-mkconfig */
	static void
	grub2_child_setup (gpointer user_data)
	{
	Grub2ChildSetupData *cdata = user_data;
	setenv ("_OSTREE_GRUB2_BOOTVERSION", cdata->bootversion_str, TRUE);
	/* We have to pass our state (whether or not we're using EFI) to the child */
	if (cdata->is_efi)
	setenv ("_OSTREE_GRUB2_IS_EFI", "1", TRUE);
	/* Everything below this is dealing with the chroot case; if
	* we're not doing that, return early.
	*/
	if (!cdata->root)
	return;
	/* TODO: investigate replacing this with bwrap */
	if (chdir (cdata->root) != 0)
	{
	perror ("chdir");
	_exit (1);
	}
	if (unshare (CLONE_NEWNS) != 0)
	{
	perror ("CLONE_NEWNS");
	_exit (1);
	}
	if (mount (NULL, "/", "none", MS_REC|MS_PRIVATE, NULL) < 0)
	{
	perror ("Failed to make / a private mount");
	_exit (1);
	}
	if (mount (".", ".", NULL, MS_BIND | MS_PRIVATE, NULL) < 0)
	{
	perror ("mount (MS_BIND)");
	_exit (1);
	}
	if (mount (cdata->root, "/", NULL, MS_MOVE, NULL) < 0)
	{
	perror ("failed to MS_MOVE to /");
	_exit (1);
	}
	if (chroot (".") != 0)
	{
	perror ("chroot");
	_exit (1);
	}
	}

expects grub-mkconfig to mount every filesystem it requires.

[cgwalters]

So I'm increasingly thinking that the bootloader backends all were a mistake, and the way things should work is via having separate systemd units that hook into ostree-finalize-staged.service. The only case where this falls over is when we're operating on an alternative root - as in creating an initial deployment as part of a disk image. But maybe for that we ask people to do it in external tooling.

The appeal here is that it severs ostree from needing to know about specific bootloaders at all. And systemd offers tons of controls and tooling for units - it's effectively already a plugin system today. We'd just have to stabilize the existence of ostree-finalize-staged.service, but we already have things like greenboot hooking into it.

[bauen1]

So ostree would start the ostree-finalize-staged.service when not using --stage instead of doing it internally then ?

I don't particularly like this idea, because it would hide errors in systemd logs, but it works, so I could live with it.

Sorry, I'm pretty sure once I implement SecureBoot with my own PK, and TPM2.0 Attestation, I will manage to break a few more assumptions ostree makes about bootloaders 😅

[ericcurtin]

aboot is similar to zipl backend in that, the main reason why having a backend is useful in this case is to flash boot data at the end of the deploy in the "post_bls_sync" stage. It's to execute final steps before next boot that an ostree commit simply cannot apply (as @cgwalters described in a different way).

There's actually a little demand for an Asahi Fedora Silverblue Remix now, if somebody actually implemented that, they might need similar hooks also, if you wanted to handle all kinds of upgrades in rpm-ostree, you also have the option of doing some upgrades outside of rpm-ostree like fwupd would do for example.

ostree-finalize-staged.service @cgwalters described could achieve the same goal in a less monolithic way, not requiring contributions directly to ostree everytime (and not necessarily requiring C code either).

[jlebon]

So I'm increasingly thinking that the bootloader backends all were a mistake, and the way things should work is via having separate systemd units that hook into ostree-finalize-staged.service. The only case where this falls over is when we're operating on an alternative root - as in creating an initial deployment as part of a disk image. But maybe for that we ask people to do it in external tooling.

The appeal here is that it severs ostree from needing to know about specific bootloaders at all. And systemd offers tons of controls and tooling for units - it's effectively already a plugin system today. We'd just have to stabilize the existence of ostree-finalize-staged.service, but we already have things like greenboot hooking into it.

Right, the issue with this I think which @bauen1 alluded to is that this only helps with the finalization flow at shutdown. I think whatever we come up with should work even if you direcly do ostree admin finalize-staged directly (or a more convincing argument where no staging is involved at all: rpm-ostree cleanup, which insta-writes the bootloader, or rpm-ostree rollback which could insta-write the bootloader).

[cgwalters]

rpm-ostree cleanup, which insta-writes the bootloader, or rpm-ostree rollback which could insta-write the bootloader).

Yes, but we control those things, and can also force-invoke ostree-finalize-staged.service which would be generalized to "synchronize external bootloader state" even if we're not shutting down.

[jlebon]

Definitely. It just feels like an awkward API vs just e.g. a dropin directory with executables (probably want both /usr and /etc variants).

[jlebon]

Here's a thorny example: you can have a staged deployment and a rollback deployment at the same time. If you just fire off ostree-finalize-staged.service on bootloader updates, rpm-ostree cleanup -r would also inadvertently finalize the staged deployment. So at the very least, I'd make it a separate unit instead, which ostree-finalize-staged.service would pull in.

But also, IMO I think systemd units which can be "re-activated" are just confusing. You almost always want RemainAfterExit=yes, but not in this case. Race conditions could also be an issue; ostree would probably want to wait until the service finishes before releasing the lock.

[jlebon]

Actually, we already have the /ostree/deploy API which can be used for this too. Basically:

root@cosa-devsh:~# systemctl cat foobar.path
# /etc/systemd/system/foobar.path
[Path]
PathChanged=/ostree/deploy

root@cosa-devsh:~# systemctl cat foobar.service
# /etc/systemd/system/foobar.service
[Service]
Type=oneshot
ExecStart=echo foobar

root@cosa-devsh:~# systemctl start foobar.path

root@cosa-devsh:~# rpm-ostree kargs --append foobar
Staging deployment... done
Changes queued for next boot. Run "systemctl reboot" to start a reboot

root@cosa-devsh:~# journalctl -u foobar.service
May 28 19:16:08 cosa-devsh systemd[1]: Starting foobar.service...
May 28 19:16:08 cosa-devsh echo[2297]: foobar
May 28 19:16:08 cosa-devsh systemd[1]: foobar.service: Deactivated successfully.
May 28 19:16:08 cosa-devsh systemd[1]: Finished foobar.service.

Two things to note:

1. This also triggers on some non-bootloader update events (e.g. staging, or unlocking a deployment). I guess it'd work to target PathChanged=/boot but that might also have false positives. Or we can also just add a new mtime bump semantic on the libostree side too (e.g. on /boot/ostree). Depending on the plugin, it might be appropriate for it to do its own change detection on top anyway if there are cases where a bootloader update doesn't actually imply a need to change anything.
2. This doesn't use RemainAfterExit=yes because we actually do want it to trigger multiple times. As mentioned above, this can be confusing I think and we'd also want to get the locking semantics right.