Switch to signed images

[castrojo]

Filing for future consideration. Currently rebasing to the signed image fails:

rpm-ostree rebase --experimental ostree-image-signed:docker://ghcr.io/ublue-os/base:latest
Pulling manifest: ostree-image-signed:docker://ghcr.io/ublue-os/base:latest
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage

According to the manpage we could set up the policy.json and probably also config in containers-registries.d to make it all seamless.

• https://www.mankier.com/5/containers-policy.json#Policy_Requirements-sigstoreSigned
• https://www.mankier.com/5/containers-registries.d

[ahgencer]

I was able to implement this in the testing branch of my own repository here. However, it didn't quite work.

The signature of the image is verified the way you'd expect with podman pull. Unsigned images, as well as images with the wrong signature are rejected; and only properly verified images are pulled.

The problem is with using the signed image with rpm-ostree. I built an image to have containers-policy.json(5) and containers-registried.d(5) configured OOTB with the public key. The idea was that I could rebase to ostree-unverified-registry:<IMAGE> first, to get the configuration in /etc set up properly; and then from there rebase again to ostree-image-signed:<IMAGE>.

While both rebases worked just fine, which gets past the problem above, the image signature does not seem to be getting verified at all. It neither rejected rebasing onto a non-signed image, nor onto an image with the wrong signature. This goes against the whole point of having the image signed in the first place, no?

[ahgencer]

After playing with this a bit further, combing through documentation, and even trying to find the relevant pieces of source code in ostreedev/ostree-rs-ext, I still wasn't able to make any progress. Also, a search for the term "ostree-image-signed" barely yields any results

Thinking there might be a bug in rpm-ostree, I filed coreos/rpm-ostree#4272 for now.

[castrojo]

Thanks for your investigation into this! I've subbed to the rpm-ostree issue!

[castrojo]

Alright this is live in skope 1.13 now, here are the best instructions/examples I can find:

• Image signature not getting verified for container-native builds signed with Cosign coreos/rpm-ostree#4272

[castrojo]

Alright! I was able to test this successfully using the examples provided but using our cosign key:

[jorge@exo ~]$ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx:latest
                   Digest: sha256:bec067c33c7f4fa6b30da124a31f6f2b7bbdd59d1fb590d1d66219ea5a8bfb05
                  Version: 38.20230716.0 (2023-07-16T10:11:21Z)
            LocalPackages: 1password-8.10.8-1.x86_64

  ostree-unverified-registry:ghcr.io/ublue-os/bluefin-dx:latest
                   Digest: sha256:bec067c33c7f4fa6b30da124a31f6f2b7bbdd59d1fb590d1d66219ea5a8bfb05
                  Version: 38.20230716.0 (2023-07-16T10:11:21Z)
            LocalPackages: 1password-8.10.8-1.x86_64

It looks like we'll want to set the type to reject and then explicitly allow "ghcr.io/ublue-os": [{"type": "insecureAcceptAnything"}], in the policy.json file?

I also wonder if there's a way we could avoid having to list every image in the yaml file?

[ahgencer]

I think setting "ghcr.io/ublue-os": [{"type": "insecureAcceptAnything"}] will tell skopeo to reject every image without a signature except ghcr.io/ublue-os, which it will accept no matter what. This is the exact opposite of what we want.

How I had done it in my repository was specify a key path specifically for the image, and then have a less specific catch-all case ("") to accept other images without a signature.

I vaguely remember this set-up working when pulling an image with podman, but rpm-ostree was not verifying signatures at the time (which is why I submitted the bug report you mentioned above). I have not tested if it works now. It would be great if you could check it out, though my test images are quite out of date by now and don't contain the necessary updates to skopeo.

[castrojo]

Yeah just noticed that when I tried to update a distrobox:

- Creating ubuntu... 
Trying to pull ghcr.io/ublue-os/ubuntu-toolbox:latest...
Error: Source image rejected: Running image docker://ghcr.io/ublue-os/ubuntu-toolbox:latest is rejected by policy.

I'll keep investigating, thanks for your help!

EDIT: adding a link to ahgencer's PR: ahgencer/ocitree@main...pr-cosign

[kth8]

I currently have my policy.json as:

{
  "default": [
    {
      "type": "reject"
    }
  ],
  "transports": {
    "docker": {
      "registry.access.redhat.com": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ],
      "registry.redhat.io": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ],
      "ghcr.io/ublue-os": [
        {
          "type": "sigstoreSigned",
          "keyPath": "/etc/pki/containers/ublue-os.pub",
          "signedIdentity": {
            "type": "matchRepository"
          }
        }
      ],
      "ghcr.io/kth8": [
        {
          "type": "sigstoreSigned",
          "keyPath": "/etc/pki/containers/kth8.pub",
          "signedIdentity": {
            "type": "matchRepository"
          }
        }
      ],
      "docker.io": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "quay.io": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "gcr.io": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "cgr.dev": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.fedoraproject.org": [
        {
          "type": "insecureAcceptAnything"
        }
      ]
    },
    "docker-daemon": {
      "": [
        {
          "type": "insecureAcceptAnything"
        }
      ]
    }
  }
}

which allows me to use signed images from ghcr.io/ublue-os and my personal image

State: idle
Deployments:
● ostree-image-signed:docker://ghcr.io/kth8/myblue
                   Digest: sha256:c638b911663c197b23fa84309a3175f1a8f3b5bbd1072815c25a5482c6b659d7
                  Version: 38.20230716.0 (2023-07-17T01:17:15Z)

  ostree-image-signed:docker://ghcr.io/ublue-os/silverblue-main
                   Digest: sha256:43810063ed47ed05f52badab6767489af4972c87353a68042d439117fcb60e70
                  Version: 38.20230716.0 (2023-07-16T07:09:57Z)

while being able to pull any unsigned images from other repos like docker.io, quay.io, etc. Tested by running

podman pull ghcr.io/ublue-os/ubuntu-toolbox
podman pull docker.io/library/alpine
podman pull quay.io/toolbx-images/alpine-toolbox:edge
podman pull gcr.io/etcd-development/etcd:v3.5.9
podman pull cgr.dev/chainguard/wolfi-base
podman pull registry.fedoraproject.org/fedora-minimal

[castrojo]

Alright thanks everyone, I got it working locally.

Last question, should we keep this in /etc or should we ship it in /usr/etc?

[akdev1l]

/usr/etc is the correct place

[castrojo]

Initial PR here, needs review and probably some RPM thing:

ublue-os/config#76

[castrojo]

Ok so I can't switch back to an unsigned image ...

rpm-ostree rebase ostree-verified-registry:ghcr.io/ublue-os/bluefin-dx
error: While pulling ghcr.io/ublue-os/bluefin-dx: Remote "ostree-verified-registry" not found

Do we know how to test to see if the change affects existing setups? I just want to doublecheck that we're not breaking updates for people not on signed images, which is basically everyone, heh.

[castrojo]

Alright, this is live, all the docs are PRed, I've swapped over and you can rebase back to fedora or another unsigned image just fine, thanks so much folks! Feel free to file follow on PRs or issues if you run into something, thanks!