hack: workaround for bootc

[ralphbean]

Do not merge. This is a temporary pipeline to for rpm-ostree images.

The image update here pulls in this change:
konflux-ci/release-service-utils#311

It makes it so that in the event that the layer data is absent, a null value won't be sent to pyxis which would be rejected by pyxis' input validation.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[mmalina]

@ralphbean I looked at https://github.com/konflux-ci/release-service-utils/pull/311/files and it adds a check for null values. But based on the conversation in Slack https://redhat-internal.slack.com/archives/C04PZ7H0VA8/p1731348500504399 it seemed there was a duplicate value (because the same layer was present in multiple images). Can you explain?

[ralphbean]

Yes - take a look at registry.redhat.io/rhel9/rhel-bootc:9.5. It's a multiarch image, with three images, one per arch:

❯ skopeo inspect --raw docker://registry.redhat.io/rhel9/rhel-bootc:9.5 | jq -r '.manifests[].digest'
sha256:d15605cc5322e04e51338af9c82dff9adc4a72633e1ae5b37985373ab33873f0
sha256:1dcae0182061678c401411efe18d87da2b3abc2961f8419cd95094114e636f42
sha256:be19744979b75ffc629aa6e38e4de87f582c37f91e9acfbfd81e32d15a2b636a

Now, let's loop over those, and for each one, inspect the top_layer_id:

❯ for archful_digest in $(skopeo inspect --raw docker://registry.redhat.io/rhel9/rhel-bootc:9.5 | jq -r '.manifests[].digest'); do 
   skopeo inspect --raw docker://registry.redhat.io/rhel9/rhel-bootc@$archful_digest | jq -r '.layers[-1].digest';
done
sha256:bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d
sha256:bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d
sha256:bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d

They're the same! That's the problem. In our task, we insert them to pyxis one after another (one arch, then the second, then the third). The first one succeeds, but the second one fails. It is rejected by pyxis, because it violates a duplicate key constraint: no two images in pyxis are allowed to have the same top_layer_id.

So, the workaround here is just to not upload that.

The reason they are duplicates is that this particular image is not squashed at build time. The last layer during build is something static that never varies. If it were squashed, then the digests would be unique.

We need a better long-term fix than this workaround.

Some options:

• We start squashing the image at build time.
• We modify pyxis to permit duplicate top layer ids. What side-effects would this have on (for instance) preflight and partner certification?
• We modify our task to only upload top_layer_id in some cases, when we know the image is a problematic one like this.

[mmalina]

FYI, I created a Jira to add a flag for this: https://issues.redhat.com/browse/RELEASE-1336

[mmalina]

And now there is a PR: #730

[openshift-merge-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.