OStree assurance when repo is untrusted

[lws-team]

Thanks for your work on ostree. I'm using ostree 2021.6 (from yocto kirkstone) in an embedded device where the rootfs partition, so also the local non-archive repo, is untrusted.

A "bootloader" earlier in the chain of trust, which has ostree available and has no dependencies on the untrusted rootfs, wants to make sure the untrusted rootfs partition represents the checkout of a signed commit before starting it Both archive and non-archive repos may be used as the source, in both cases, the source repo and the destination rootfs may have been tampered, but ostree itself and the trusted pubkey for signed commit checking can be assumed untampered.

What is the assurance ostree can actually give about that?

• Be able to always confirm the signed commit signature against the gpg pubkey and fatal error if fails?
• During checkout, confirm the hashes on files from the source repo (when decompressed) match the signed commit, and confirm hashes on extant files match the signed commit such that if the checkout completes, what is in the checkout dir fully represents the content of the signed commit?

I ask because I was very confused after updating the rootfs /boot directly with development kernel updates via scp. This worked but it seemed because the local repo shared an inode with the checkout file, when the checked-out file was overwritten, so was the uncompressed file in the repo. I could no longer use ostree to revert the rootfs kernel to the one in the signed checkout, it acted like all was well. but it brought out the changed kernel from the repo. So there are two problems there:

1. If I want to allow this (development flow with scp overwriting checked-out files), I should not use hardlinks (I am trying -C)

2. It seems there is no actual assurance of anything coming from the signed commit. Ostree did not seem to take any care about enforcing what was checked out matched the signed commit.

Am I wrong with my model of what is happening, or missing a trick or this is expected? If it's expected, please help me understand the limits of what it means to have a "signed commit" for ostree.

[cgwalters]

Hi, you're right that there is no dynamic runtime verification. This gap is expected to be plugged by https://github.com/containers/composefs

That said, one thing we could do is add a flag to ostree admin deploy (and the API) that requires that the commit was signed, and that the signature was verified. Today the signature verification status is not persistently stored, but it probably should be so that one can at least reliably detect that the signature was verified at the time of pull.

[cgwalters]

At runtime, the read-only bind mount protects things. I did also experiment with support for fs-verity, which even without signatures disallows all writes to file content. If your root filesystem has fs-verity support, you could consider that. Enabling it (even without composefs) significantly strengthens dynamic runtime protection.

[lws-team]

Thanks for the reply.

I think the issue boils down that current ostree security model trusts the rootfs partition, where the repo contents and the deployment are stored, same as you trust your own laptop. Validating sigs at pull time fits into this model that once it's on there, you trust it won't get tampered before you deploy it. From this perspective, a ro bind mount 'protects' the repo from modification.

In the case the security model is you can't trust the rootfs, and it may by one means or another be modified at will by the attacker, validating sigs at pull time only doesn't do anything for security. From this perspective, ro bind mounts won't protect anything if the underlying storage can be tampered.

If I understood the problem with the repo object overwrite via hardlink correctly (apologies if I got the wrong idea...), it isn't confirming object hash contents against the signed commit either... if you don't trust the rootfs then there is no reason to simply believe whatever you took from ./objects/ab/cdef012.... still has the hash abcdef012... and isn't tampered. Bringing the 'right' object out of the object store unchecked, even at the behest of a signed commit, isn't adding any security in that case since the repo's file you get for the right, signed hash may have been changed to anything with any different hash.

It's true that sig checking from the untrusted rootfs using keys from untrusted or tools from untrusted won't help. But this particular system has a chain-of-trust initramfs with trusted pubkeys and tools in it. So it could solve it by confirming the hashes on the rootfs match those on the confirmed-valid signed commit, forcing any that don't match and deleting what shouldn't be there, before jumping into init.

[cgwalters]

If you're OK with the boot-time latency, today ostree fsck --delete from the initramfs would detect modifications to the repository, although not the deployment directories (we are missing a fsck for this).

But I think few people want that latency, so the composefs (backed by fs-verity) is what we've been looking at most recently.

[lws-team]

It's the state of the deployed rootfs that matters, not the repo, so long as the deployed files' hashes are confirmed with the signed - and verified - commit details it will uncover tampering at the repo.

Thanks for the links to fs-verity and composefs, I'll look closely at it tomorrow.

[cgwalters]

Going to duplicate this against #2867