Multiple concurrent invocations of the attest command can result in lost attestations

[johanngyger]

Running cosign attest <image> (almost) concurrently can have the side effect that attestations written to the container registry previously are overridden by later invocations:

1. cosign attest no 1 reads existing attestations
2. cosign attest no 2 reads existing attestations
3. cosign attest no 1 writes attestations with new attestation1
4. cosign attest no 2 writes attestations with new attestation2 and the effect that attestation1 is lost

[znewman01]

This proposed change to the OCI distribution spec would help: opencontainers/distribution-spec#250

We could hack around this with some terrible locking implemented over OCI. @imjasonh @jdolitsky any precedent for that in OCI?

[imjasonh]

That OCI spec change was proposed more or less to solve exactly this issue, for cosign specifically. The same can happen if two process are cosign signing the same image.

I think we might be close to having OCI specify the recommendation of using ETags for this, but it won't necessarily be enforced by all registries. The concern in OCI has been that some legacy registries use backing storage that's fundamentally eventually consistent (e.g., old AWS S3 buckets), so some registries may not be able to enforce an ETag. So folks will want a way to know whether their ETag was honored or not, so clients like cosign can fail if a race condition can't be ruled out.

In any case I'd like OCI and registries to be responsible for solving this in an HTTP-standard way like ETags, rather than bolt on locking into cosign and any other tools.

[blairdrummond]

I just want to double-check my understanding as a consumer here:

Is the current state?:

This race-condition is real, but ETag support can't be universally supported by registries so solving this concurrency issue generically isn't feasible.

Instead, I should look into the new Referrers API in OCI 1.1, and just avoid this concurrency issue entirely by not trying to write to the same tag while uploading attestations? (noting that cosign doesn't currently support that for attestations, only deprecated SBOMs, but it likely will fix that soon)