podman push --compression-format=zstd:chunked generates invalid OCI images

[Romain-Geissler-1A]

Issue Description

It seems that --compression-format=zstd:chunked generates invalid OCI images. When pushed in a registry, and then pull by docker, docker complains with "layers from manifest don't match image configuration".

Steps to reproduce the issue

Here is how to reproduce using the very latest podman upstream image:

• Start a container with the very latest podman:

> podman run -t -i --rm --privileged --pull=always quay.io/podman/upstream 
Trying to pull quay.io/podman/upstream:latest...
...
Writing manifest to image destination

• Install some simple dependencies for investigation

[root@2cc5686c4046 /]# dnf install -y vim less zstd jq file golang-github-vbatts-tar-split

• Pull a fedora image

[root@2cc5686c4046 /]# podman pull quay.io/fedora/fedora
Trying to pull quay.io/fedora/fedora:latest...
Getting image source signatures
Copying blob 718a00fe3212 done   |
Copying config 368a084ba1 done   |
Writing manifest to image destination
368a084ba17dcba88f5b23acfa47481131010219524fd9c41af87d709a04845b

• Now export this image in 3 different compression format on disk:

[root@2cc5686c4046 /]# podman push --compression-format=gzip quay.io/fedora/fedora oci:image.gzip
Getting image source signatures
Copying blob 8ff7ad910417 done   |
Copying config 3647899ed3 done   |
Writing manifest to image destination
[root@2cc5686c4046 /]# podman push --compression-format=zstd quay.io/fedora/fedora oci:image.zstd
Getting image source signatures
Copying blob 8ff7ad910417 done   |
Copying config 3647899ed3 done   |
Writing manifest to image destination
[root@2cc5686c4046 /]# podman push --compression-format=zstd:chunked quay.io/fedora/fedora oci:image.zstd-chunked
Getting image source signatures
Copying blob 8ff7ad910417 done   |
Copying config 3647899ed3 done   |
Writing manifest to image destination

• Now let's compare the digest of the "config" artefact of each docker OCI image. We use jq to do a bit of indirection and read the right blob file. The digest is the same for all 3 OCI images, as expected. At the end, we show the rootfs of this config file (from the gzip image since it's the same on all images):

[root@2cc5686c4046 /]# jq ".config.digest" "image.gzip/blobs/$(jq -r ".manifests[0].digest" image.gzip/index.json | tr : /)"
"sha256:3647899ed34698fd935c6e6631b24a76c6213e9f3df4380eb371b0836b4287d9"
[root@2cc5686c4046 /]# jq ".config.digest" "image.zstd/blobs/$(jq -r ".manifests[0].digest" image.zstd/index.json | tr : /)"
"sha256:3647899ed34698fd935c6e6631b24a76c6213e9f3df4380eb371b0836b4287d9"
[root@2cc5686c4046 /]# jq ".config.digest" "image.zstd-chunked/blobs/$(jq -r ".manifests[0].digest" image.zstd-chunked/index.json | tr : /)"
"sha256:3647899ed34698fd935c6e6631b24a76c6213e9f3df4380eb371b0836b4287d9"

[root@2cc5686c4046 /]# jq ".rootfs" "image.gzip/blobs/$(jq -r ".config.digest" "image.gzip/blobs/$(jq -r ".manifests[0].digest" image.gzip/index.json | tr : /)" | tr : /)"
{
  "type": "layers",
  "diff_ids": [
    "sha256:8ff7ad910417a7b8a49019008335921d2aac0e3304a19ce258deabf431e59801"
  ]
}

• Now let's do like what docker does, and compute the actual sha256 sum of the uncompressed layer tarball. For gzip and zstd the sha256 sum is equal, and matches the config's rootfs DiffId. However for zstd:chunked, it doesn't match.

[root@2cc5686c4046 /]# zcat "image.gzip/blobs/$(jq -r ".layers[0].digest" "image.gzip/blobs/$(jq -r ".manifests[0].digest" image.gzip/index.json | tr : /)" | tr : /)" | file -
/dev/stdin: POSIX tar archive (GNU)
[root@2cc5686c4046 /]# zcat "image.gzip/blobs/$(jq -r ".layers[0].digest" "image.gzip/blobs/$(jq -r ".manifests[0].digest" image.gzip/index.json | tr : /)" | tr : /)" | sha256sum
8ff7ad910417a7b8a49019008335921d2aac0e3304a19ce258deabf431e59801  -
[root@2cc5686c4046 /]#
[root@2cc5686c4046 /]# zstdcat "image.zstd/blobs/$(jq -r ".layers[0].digest" "image.zstd/blobs/$(jq -r ".manifests[0].digest" image.zstd/index.json | tr : /)" | tr : /)" | file -
/dev/stdin: POSIX tar archive (GNU)
[root@2cc5686c4046 /]# zstdcat "image.zstd/blobs/$(jq -r ".layers[0].digest" "image.zstd/blobs/$(jq -r ".manifests[0].digest" image.zstd/index.json | tr : /)" | tr : /)" | sha256sum
8ff7ad910417a7b8a49019008335921d2aac0e3304a19ce258deabf431e59801  -
[root@2cc5686c4046 /]# zstdcat "image.zstd-chunked/blobs/$(jq -r ".layers[0].digest" "image.zstd-chunked/blobs/$(jq -r ".manifests[0].digest" image.zstd-chunked/index.json | tr : /)" | tr : /)" | file - 
/dev/stdin: POSIX tar archive (GNU)
[root@2cc5686c4046 /]# zstdcat "image.zstd-chunked/blobs/$(jq -r ".layers[0].digest" "image.zstd-chunked/blobs/$(jq -r ".manifests[0].digest" image.zstd-chunked/index.json | tr : /)" | tr : /)" | sha256sum 
57b0ecf19f5d86d4002f7998b1f336f026d2c6301f74463234a76712d8d753a2  -

From what I understood of zstd:chunked, the idea was that the metadata would be added as annotation in the OCI image config, rather than inside hidden tar metadata (ie unlinke estargz). But it seems the generated tarball in the end is not the same.

• Checking some basic "diff" at basic metadata level between zstd and zstd:chunked, I don't see any differences:

[root@2cc5686c4046 /]# diff -u <(zstdcat "image.zstd/blobs/$(jq -r ".layers[0].digest" "image.zstd/blobs/$(jq -r ".manifests[0].digest" image.zstd/index.json | tr : /)" | tr : /)" | tar -v -t) <(zstdcat "image.zstd-chunked/blobs/$(jq -r ".layers[0].digest" "image.zstd-chunked/blobs/$(jq -r ".manifests[0].digest" image.zstd-chunked/index.json | tr : /)" | tr : /)" | tar -v -t)

• Now let's use tar-split (https://github.com/vbatts/tar-split from your Docker friends ;) ) to investigate a bit more the diffs at tar level:

[root@2cc5686c4046 /]# tar-split disasm --output tar-data.zstd.json.gz <(zstdcat "image.zstd/blobs/$(jq -r ".layers[0].digest" "image.zstd/blobs/$(jq -r ".manifests[0].digest" image.zstd/index.json | tr : /)" | tr : /)") | sha256sum
INFO[0000] created tar-data.zstd.json.gz from /dev/fd/63 (read 182886400 bytes) 
8ff7ad910417a7b8a49019008335921d2aac0e3304a19ce258deabf431e59801  -
[root@2cc5686c4046 /]# tar-split disasm --output tar-data.zstd-chunked.json.gz <(zstdcat "image.zstd-chunked/blobs/$(jq -r ".layers[0].digest" "image.zstd-chunked/blobs/$(jq -r ".manifests[0].digest" image.zstd-chunked/index.json | tr : /)" | tr : /)") | sha256sum
INFO[0000] created tar-data.zstd-chunked.json.gz from /dev/fd/63 (read 182880256 bytes) 
57b0ecf19f5d86d4002f7998b1f336f026d2c6301f74463234a76712d8d753a2  -



[root@2cc5686c4046 /]# diff -u <(zcat tar-data.zstd.json.gz) <(zcat tar-data.zstd-chunked.json.gz)
--- /dev/fd/63  2023-12-05 23:44:13.901880393 +0000
+++ /dev/fd/62  2023-12-05 23:44:13.901880393 +0000
@@ -16803,5 +16803,4 @@
 {"type":2,"payload":"Li9saWI2NAAAAAA (many AAAA ommitted) AAAAADAwMDA3NzcAMDAwMDAwMAAwMDAwMDAwA
DAwMDAwMDAwMDAwADE0NDU2MzQ1MjAwADAxMjIzMAAgMnVzci9saWI2NAAAAAAAA (many AAAA ommitted)  AAAAAB1c3RhciAg
AHJvb3QAAAAAAAAAAA(many AAAA ommitted) AAAAAAAAAAcm9vdAAAAAAAAAA (many AAAA ommitted) AAAAAAAAAA=","position":16802}
 {"type":1,"name":"./lib64","payload":null,"position":16803}
 {"type":2,"payload":"AAAAAAAAAAAAAAAA (many AAAA ommitted) AAAAAAAAAAAAA==","position":16804}
-{"type":2,"payload":"AAAAAAAAAAAA (many AAAA omitted) AAAAAAAAAAAAAAAAA","position":16805} 
-{"type":2,"payload":"","position":16806} 
+{"type":2,"payload":"","position":16805} 

I am not sure if I shall continue the investigation myself from there or if you want to continue on your side ;) But it seems there is some unexpected behavior with zstd:chunked compression.

Describe the results you received

The generated layer tarball in zstd:chunked is not correct, while it is for gzip/zstd.

Describe the results you expected

The generated layer tarball in should be the same for all compression, including zstd:chunked.

podman info output

Tried with the latest `quay.io/podman/upstream`.

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[Romain-Geissler-1A]

Note that actually only after I finished writing this investigation I have found this issue which seems to deal with some similar concerns: containers/podman#20611 So maybe my issue is just a duplicate of this known issue.

[rhatdan]

@giuseppe PTAL

[giuseppe]

I'll give it a try but this should be fixed with: containers/image#1980

[Romain-Geissler-1A]

Ok thanks. As soon as this is vendored into podman and released into a quay.io/podman/upstream image, I can test as well the full flow on my side, with pushing to a registry and re-pulling back on docker side.

Note that in another scenario where I would like to adopt zstd:chunked, I would like to build images via buildx (connecting to a real docker daemon, as this is the only thing I have in Jenkins environment for now), export it to an oci-archive via buildx, then push it to a registry and re-compressing it on the fly to zstd:chunked via skopeo. And so far my attempt to create zstd:chunked image this way via skopeo instead of podman failed similarly, so I hope your fix will apply to skopeo too ;)

[giuseppe]

it seems still broken with the current version, taking a look now

[giuseppe]

opened a PR: #1772