Docker Containerd snapshotter Overlayfs, docker save fails for image pulled using --platform=linux/arm64 on amd64 host. #5476

chetanshivaji · 2024-09-24T12:23:03Z

Description

docker save fails for image pulled using --platform=linux/arm64 on amd64 host.

I pulled image as,
docker pull --platform=linux/arm64 nginx:latest

docker images --no-trunc
REPOSITORY   TAG       IMAGE ID                                                                  CREATED       SIZE
nginx        latest    sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3   5 weeks ago   273MB

docker save -o nginx.tar sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3**
Error response from daemon: unable to create manifests file: content digest sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e: not found

Not able to do docker save because of above error

Extra info,
Although, docker save works with arm64 arch image if image is pulled using mainfest digest as below,

docker pull nginx:latest@sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171

docker images --no-trunc
REPOSITORY   TAG       IMAGE ID                                                                  CREATED       SIZE
nginx        <none>    sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171   5 weeks ago   273MB


 docker save -o nginx.tar sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171 - success

Reproduce

Same as above description.

Expected behavior

No response

docker version

docker version
Client:
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:00 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:13:08 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.8
  GitCommit:        7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[root@localhost sha256]#

docker info

# docker info
Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc.)
    Version:  v0.8.0
    Path:     /usr/libexec/docker/cli-plugins/docker-app
  buildx: Build with BuildKit (Docker Inc.)
    Version:  v0.3.1-tp-docker
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 25.0.3
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.4.17-2136.323.8.2.el7uek.x86_64
 Operating System: Oracle Linux Server 7.9
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.544GiB
 Name: localhost.localdomain
 ID: 0844a04b-e797-4332-881c-74867bca5af0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTPS Proxy: http://10.115.27.54:3128/
 No Proxy: localhost,127.0.0.1,localaddress,.localdomain.com
 Username: [email protected]
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[root@localhost sha256]#

Additional Info

No response

The text was updated successfully, but these errors were encountered:

thaJeztah · 2024-09-24T18:50:50Z

Thanks for reporting! Yes, this is needs some work. I was curious if this was fixed on current master of the engine, and if PR would have fixed it;

But it looks like, while the error is slightly different 27.3 and "master" (looks like it now includes a NotFound prefix; possibly originating from containerd error-defs), the issue is still the same;

docker pull --platform=linux/s390x nginx:latest
latest: Pulling from library/nginx
a95d04829aec: Download complete
42b2ec246294: Download complete
95fe27c895a8: Download complete
6982f08f280c: Download complete
b2fff712f8b4: Download complete
1cdb42c01f48: Download complete
f9cf33a9c2ac: Download complete
Digest: sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

docker image ls --tree
WARNING: This is an experimental feature. The output may change and shouldn't be depended on.

IMAGE                   ID             DISK USAGE   CONTENT SIZE   IN USE
nginx:latest            04ba374043cc        246MB         65.3MB
├─ linux/s390x          0fa1ac852fea        246MB         65.3MB
├─ linux/amd64          88a0a069d5e9           0B             0B
├─ linux/arm/v5         6289301f3b91           0B             0B
├─ linux/arm/v7         d719acf8bcaf           0B             0B
├─ linux/arm64/v8       9f661996f4d1           0B             0B
├─ linux/386            67e126c43a8d           0B             0B
├─ linux/mips64le       83277e45a916           0B             0B
└─ linux/ppc64le        964b97f05849           0B             0B

docker save -o nginx.tar nginx:latest
Error response from daemon: unable to create manifests file: NotFound: content digest sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171: not found

daemon logs (current master) look like;

DEBU[2024-09-24T18:49:43.377261423Z] Calling HEAD /_ping                           spanID=c4ad8816f0c9fe7c traceID=d6ece5243ab000edd0dee3f6857257da
DEBU[2024-09-24T18:49:43.378616590Z] Calling GET /v1.48/images/get?names=nginx%3Alatest  spanID=0d7b5e44f6d43c8a traceID=d9a86fa6439b281f1f9320fbb2ffdcf8
DEBU[2024-09-24T18:49:43.385920632Z] export requested                              img="{docker.io/library/nginx:latest map[] {application/vnd.oci.image.index.v1+json sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3 10272 [] map[] [] <nil> } 2024-09-24 18:46:01.821534085 +0000 UTC 2024-09-24 18:46:01.821534085 +0000 UTC}" name="nginx:latest" resolveErr="<nil>" spanID=0d7b5e44f6d43c8a specificDigestResolved=false traceID=d9a86fa6439b281f1f9320fbb2ffdcf8
DEBU[2024-09-24T18:49:43.401433173Z] export image                                  name="docker.io/library/nginx:latest" spanID=0d7b5e44f6d43c8a target="{application/vnd.oci.image.index.v1+json sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3 10272 [] map[] [] <nil> }" traceID=d9a86fa6439b281f1f9320fbb2ffdcf8

The --platform flag (from #5331) helps in this case, but either the error message should be better (i.e., "we have the image, but not in the default platform"), or it should export only the available one;

docker save -o nginx.tar --platform=linux/s390x nginx:latest

ls -la nginx.tar
-rw------- 1 root root 65330176 Sep 24 18:47 nginx.tar

thaJeztah · 2024-09-24T19:00:08Z

Also tried to find what the sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171 digest refers to, and it looks indeed the default (arm64) image;

docker buildx imagetools inspect nginx:latest
...
...
  Name:        docker.io/library/nginx:latest@sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64/v8
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: arm64v8
    org.opencontainers.image.base.digest:     sha256:ace984fea4de21d4ad6d0047c3c6ab7ade9fcd9dee3e44c5fe6349d7dd063d86
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T12:38:34Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian

Full output;

docker buildx imagetools inspect nginx:latest
Name:      docker.io/library/nginx:latest
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3

Manifests:
  Name:        docker.io/library/nginx:latest@sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64
  Annotations:
    org.opencontainers.image.created:         2024-09-04T23:10:40Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: amd64
    org.opencontainers.image.base.digest:     sha256:903d3225acecaa272bbdd7273c6c312c2af8b73644058838d23a8c9e6e5c82cf
    org.opencontainers.image.base.name:       debian:bookworm-slim

  Name:        docker.io/library/nginx:latest@sha256:a6c3d2371006c10c41f4d3be8db4561b8f21c7cda4e8a3ab464521a00c336826
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: amd64
    vnd.docker.reference.digest:              sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e
    vnd.docker.reference.type:                attestation-manifest

  Name:        docker.io/library/nginx:latest@sha256:6289301f3b918b98577fe0f80c13761aae5791b0059ed9e1d554edeae7c54b82
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm/v5
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.0
    com.docker.official-images.bashbrew.arch: arm32v5
    org.opencontainers.image.base.digest:     sha256:aef22c7b68e2f7707fabdff982d4d4928ca261775bae0ddb7f7f8358ed0ef10f
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-08-13T10:44:11Z
    org.opencontainers.image.revision:        a6f7d140744f8b15ff4314b8718b3f022efc7f43
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#a6f7d140744f8b15ff4314b8718b3f022efc7f43:mainline/debian

  Name:        docker.io/library/nginx:latest@sha256:3e576b43defa219d279961a042477c8df1b2e46d15ec8cd98827eae750e33fa6
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: arm32v5
    vnd.docker.reference.digest:              sha256:6289301f3b918b98577fe0f80c13761aae5791b0059ed9e1d554edeae7c54b82
    vnd.docker.reference.type:                attestation-manifest

  Name:        docker.io/library/nginx:latest@sha256:d719acf8bcaf35a0a81288ea2a60a0fcab0b4cd65a0b6f68cadb060c5861f16e
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm/v7
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: arm32v7
    org.opencontainers.image.base.digest:     sha256:756f2240fee673990dccc7bdd288792e149ce50072d5cd29d328e4cd80aa6979
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T04:28:40Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian

  Name:        docker.io/library/nginx:latest@sha256:3a254dd24b20ecb921ff418f880892562a5d11a4de7e08465ab8296e4bc6a327
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: arm32v7
    vnd.docker.reference.digest:              sha256:d719acf8bcaf35a0a81288ea2a60a0fcab0b4cd65a0b6f68cadb060c5861f16e
    vnd.docker.reference.type:                attestation-manifest

  Name:        docker.io/library/nginx:latest@sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64/v8
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: arm64v8
    org.opencontainers.image.base.digest:     sha256:ace984fea4de21d4ad6d0047c3c6ab7ade9fcd9dee3e44c5fe6349d7dd063d86
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T12:38:34Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian

  Name:        docker.io/library/nginx:latest@sha256:0df868782671055c19e022b02b723375891f278fb785c0a89b4107b815c3ca1c
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.type:                attestation-manifest
    com.docker.official-images.bashbrew.arch: arm64v8
    vnd.docker.reference.digest:              sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171

  Name:        docker.io/library/nginx:latest@sha256:67e126c43a8d738f2d05a7a7baa94a26b4d3f3f4cfb70b1aa49a52e700ef0709
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/386
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: i386
    org.opencontainers.image.base.digest:     sha256:a09e6f30093763451525093f6c1f425fcb751de9afc5c7944a28e4e48250f4c9
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T00:14:49Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian

  Name:        docker.io/library/nginx:latest@sha256:96d7658c6592eee20fcb834cc3319cb7cbdaf487614132c3ff4da35b63e3baa9
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.type:                attestation-manifest
    com.docker.official-images.bashbrew.arch: i386
    vnd.docker.reference.digest:              sha256:67e126c43a8d738f2d05a7a7baa94a26b4d3f3f4cfb70b1aa49a52e700ef0709

  Name:        docker.io/library/nginx:latest@sha256:83277e45a9163e419c5e676f23498233b4e8f283daed1174db46a6e2d7860efa
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/mips64le
  Annotations:
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T11:19:25Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: mips64le
    org.opencontainers.image.base.digest:     sha256:460b58fe46216c353ee64696b20a853678b1a08da92a64a1caf5e6faf997b329

  Name:        docker.io/library/nginx:latest@sha256:6f8fb2d834e6f7d91039dd2ddb653b809cfdb60f2646d839d17c714d8a2e2b9b
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: mips64le
    vnd.docker.reference.digest:              sha256:83277e45a9163e419c5e676f23498233b4e8f283daed1174db46a6e2d7860efa
    vnd.docker.reference.type:                attestation-manifest

  Name:        docker.io/library/nginx:latest@sha256:964b97f05849b3ca619df5bea76d98b53196260cf8cd9b6c03ccbb8393dc8613
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/ppc64le
  Annotations:
    com.docker.official-images.bashbrew.arch: ppc64le
    org.opencontainers.image.base.digest:     sha256:f39559bf405679e55f74c361f383e0cfe19b01934a53eca2149ddd966604293b
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T00:43:55Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1

  Name:        docker.io/library/nginx:latest@sha256:a72749cca16dd47485c9220a98b74b04d6f564ef8ec6c189e85c56e04c7e4dc6
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: ppc64le
    vnd.docker.reference.digest:              sha256:964b97f05849b3ca619df5bea76d98b53196260cf8cd9b6c03ccbb8393dc8613
    vnd.docker.reference.type:                attestation-manifest

  Name:        docker.io/library/nginx:latest@sha256:0fa1ac852feac85b15bdec82773c1b19eafdde054536e2f7f4c05cecbff96433
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/s390x
  Annotations:
    com.docker.official-images.bashbrew.arch: s390x
    org.opencontainers.image.base.digest:     sha256:d11dd86eb4857cdeac7b99af3884fa13e8a3d18ce550f77cee3cca64a735f233
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T02:49:36Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1

  Name:        docker.io/library/nginx:latest@sha256:7df7564b1cfdd1f098b8ffb3d91a91ed248a1d54826a8260b52ff225b70cc353
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: s390x
    vnd.docker.reference.digest:              sha256:0fa1ac852feac85b15bdec82773c1b19eafdde054536e2f7f4c05cecbff96433
    vnd.docker.reference.type:                attestation-manifest

chetanshivaji · 2024-09-25T14:30:34Z

Thanks for the details @thaJeztah.

This seems to be an issue with image pulls with --platform mismatching to host arch.

docker inspect 04ba374043cc|grep -ani arch

74:        "Architecture": "386",

docker images --no-trunc

REPOSITORY            TAG       IMAGE ID                                                                  CREATED             SIZE
nginx                 latest    sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3   5 weeks ago         252MB

docker save -o nginx386arch.tar sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3

Error response from daemon: unable to create manifests file: content digest sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e: not found

thaJeztah · 2024-09-25T15:29:11Z

@chetanshivaji yes, correct; the issue is that the image pulled does not match the default platform. When running docker save, the initial part "succeeds" (multi-arch image exists), but it then tries to save the default (native) platform, and that is missing.

So at least the error-message should be more clear, but we should also look if the request is non-ambiguous (i.e., image exists, and there's only a single platform locally -> in that case we could save the image, regardless if it's the default platform).

thaJeztah · 2024-10-03T10:22:26Z

x-posting here from #5493 (comment), because I somewhat assumed that this never worked with the containerd image store enabled, but it looks like older versions did handle this case.

With the added note that;

It's still possible that this was an intentional change in behavior (i.e., default to native platform instead of "pick whatever is found"), but at least the error-reporting should be more clear on that if that's the case.

docker version --format '{{.Server.Version}}'
# 24.0.9

docker info --format '{{ .Driver }} {{ .DriverStatus }}'
# overlayfs [[driver-type io.containerd.snapshotter.v1]]

docker pull --platform=linux/amd64 alpine

docker images
# REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
# alpine       latest    beefdbd8a1da   4 minutes ago   12.1MB

docker save -o foo.tar beefdbd8a1da

docker save beefdbd8a1da | tar -tf-
# blobs/
# blobs/sha256/
# blobs/sha256/33735bd63cf84d7e388d9f6d297d348c523c044410f553bd878c6d7829612735
# blobs/sha256/43c4264eed91be63b206e17d93e75256a6097070ce643c5e8f0379998b44f170
# blobs/sha256/91ef0af61f39ece4d6710e465df5ed6ca12112358344fd51ae6a3b886634148b
# index.json
# manifest.json
# oci-layout

thaJeztah · 2024-10-07T14:29:53Z

I think the problem may be in this area; https://github.com/moby/moby/blob/2d049d11a963e6721d1e65f22baf43ede4e518d0/daemon/containerd/image_exporter.go#L56-L67

// This makes the exported archive also include `manifest.json`
// when the image is a manifest list. It is needed for backwards
// compatibility with Docker image format.
// The containerd will choose only one manifest for the `manifest.json`.
// Our preference is to have it point to the default platform.
// Example:
//  Daemon is running on linux/arm64
//  When we export linux/amd64 and linux/arm64, manifest.json will point to linux/arm64.
//  When we export linux/amd64 only, manifest.json will point to linux/amd64.
// Note: This only matters when importing this archive into non-containerd Docker.
// Importing the same archive into containerd, will not restrict the platforms.
archive.WithPlatform(pm),

For backwards-compatiblity, it's expected to have a manifest.json, which defaults to the native platform. In this case, it may find the manifest, but it's empty (?), or at least, the list of records is non-empty, and it may now expect the native platform to be there (in this case it isn't), and fail because of that; https://github.com/moby/moby/blob/2d049d11a963e6721d1e65f22baf43ede4e518d0/vendor/github.com/containerd/containerd/images/archive/exporter.go#L336-L350

if !eo.skipDockerManifest && len(dManifests) > 0 {
    tr, err := manifestsRecord(ctx, store, dManifests)
    if err != nil {
        return fmt.Errorf("unable to create manifests file: %w", err)
    }

    records = append(records, tr)
}

if len(algorithms) > 0 {
    records = append(records, directoryRecord("blobs/", 0755))
    for alg := range algorithms {
        records = append(records, directoryRecord("blobs/"+alg+"/", 0755))
    }
}

vvoland · 2024-10-22T09:08:21Z

If the native platform is not available, the pm matcher will match any platform. The WithPlatform was specifically added to handle this situation, but looks like something changed (on our side or containerd). Will take a look.

chetanshivaji added kind/bug status/0-triage labels Sep 24, 2024

chetanshivaji changed the title ~~docker save fails for image pulled using --platform=linux/arm64 on amd64 host.~~ Docker Containerd snapshotter Overlayfs, docker save fails for image pulled using --platform=linux/arm64 on amd64 host. Sep 24, 2024

thaJeztah added containerd-integration Issues and PRs related to containerd integration version/25.0 labels Sep 24, 2024

thaJeztah added this to containerd integration Sep 24, 2024

thaJeztah moved this to To do in containerd integration Sep 24, 2024

thaJeztah added version/27.3 and removed status/0-triage labels Sep 24, 2024

thaJeztah mentioned this issue Oct 3, 2024

Containerd snapshotter - When QEMU is installed, docker save failing, for images with mismatch arch pulled with "--platform" seems Regression on 27 compared to 24.0.7 #5493

Closed

thaJeztah added this to the 28.0.0 milestone Oct 22, 2024

vvoland mentioned this issue Oct 22, 2024

c8d/save: Add tests moby/moby#48722

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Containerd snapshotter Overlayfs, docker save fails for image pulled using --platform=linux/arm64 on amd64 host. #5476

Docker Containerd snapshotter Overlayfs, docker save fails for image pulled using --platform=linux/arm64 on amd64 host. #5476

chetanshivaji commented Sep 24, 2024 •

edited by thaJeztah

Loading

thaJeztah commented Sep 24, 2024

thaJeztah commented Sep 24, 2024

chetanshivaji commented Sep 25, 2024 •

edited by thaJeztah

Loading

thaJeztah commented Sep 25, 2024

thaJeztah commented Oct 3, 2024

thaJeztah commented Oct 7, 2024

vvoland commented Oct 22, 2024

Docker Containerd snapshotter Overlayfs, docker save fails for image pulled using --platform=linux/arm64 on amd64 host. #5476

Docker Containerd snapshotter Overlayfs, docker save fails for image pulled using --platform=linux/arm64 on amd64 host. #5476

Comments

chetanshivaji commented Sep 24, 2024 • edited by thaJeztah Loading

Description

Reproduce

Expected behavior

docker version

docker info

Additional Info

thaJeztah commented Sep 24, 2024

thaJeztah commented Sep 24, 2024

chetanshivaji commented Sep 25, 2024 • edited by thaJeztah Loading

thaJeztah commented Sep 25, 2024

thaJeztah commented Oct 3, 2024

thaJeztah commented Oct 7, 2024

vvoland commented Oct 22, 2024

chetanshivaji commented Sep 24, 2024 •

edited by thaJeztah

Loading

chetanshivaji commented Sep 25, 2024 •

edited by thaJeztah

Loading