podman pull doesn't cache downloaded layers

[Siegfriedk]

Issue Description

I have currently network issues due to some macos update and when i try to download a big image with podman, it stops at some point:

Copying blob sha256:c90f61985563e49c4fa30c6db1830178ded43f1eb33a96e64936bb65b8c368de

When i cancel the pull and retry it, it is stuck, most of the time, at the same sha256.

Steps to reproduce the issue

Steps to reproduce the issue
1.
2.
3.

Describe the results you received

It gets stuck at Copying blob sha256...

Describe the results you expected

Downloading a big image on a flaky internet connection should work. Proper timeout and retry mechanism in place.

podman info output

host:
  arch: arm64
  buildahVersion: 1.35.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.64
    systemPercent: 0.16
    userPercent: 0.19
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "39"
  eventLogger: journald
  freeLocks: 2043
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.7.7-200.fc39.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 100737024
  memTotal: 2047598592
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.20240312103946045036.main.18.g8377c0a.fc39.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.11.0-dev
    package: netavark-1.10.1-1.20240319111419242122.main.46.gcc3f35d.fc39.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.11.0-dev
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.20240302220834691516.main.10.g64ee22c.fc39.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: feb70bc2ab11944a6443e4f5d2eb96a22f186b80
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.aarch64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  rootlessNetworkCmd: ""
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 50h 56m 29.00s (Approximately 2.08 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 1
    stopped: 3
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 24693854208
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 25
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.0-dev-8a643c243
  Built: 1710720000
  BuiltTime: Mon Mar 18 01:00:00 2024
  GitCommit: ""
  GoVersion: go1.21.8
  Os: linux
  OsArch: linux/arm64
  Version: 5.0.0-dev-8a643c243

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[baude]

You are using a Podman 5 pre-release binary based on the podman info provided. Also, podman does have retry logic, etc. Would it be possible to try a newer Podman version ?

[vrothberg]

Podman does in fact cache downloaded layers and will apply them to the storage as soon as possible. But a given layer must be downloaded completely for that to work.

[Siegfriedk]

You are using a Podman 5 pre-release binary based on the podman info provided. Also, podman does have retry logic, etc. Would it be possible to try a newer Podman version ?

I think i use podman through brew. I can update / change this.

Regarding your section question: i also assume thats the case because i have seen restarting the pull will be stuck at the same layer. But there has to be a better retry mechanism for one layer than just being stuck for hours (i retried it plenty of times and while doing that i also left for lunch at least once)

[vrothberg]

I think i use podman through brew. I can update / change this.

That would be great. -dev versions may have bugs which I hope is the case for you.

[baude]

@Siegfriedk fwiw, we generally recommend downloading the podman from its release binaries as we have no control over brew in the sense that it, podman, and its dependencies can change and get out of sync leading to failures. something to think about anyways ...

[Siegfriedk]

I reinstalled Podman Desktop through the website and official binary but the version is still a dev version and the bug is also still there:

APIVersion: 5.0.0-dev-8a643c243 Built: 1710720000 BuiltTime: Mon Mar 18 01:00:00 2024 GitCommit: "" GoVersion: go1.21.8 Os: linux OsArch: linux/arm64 Version: 5.0.0-dev-8a643c243

I also think that Podman Desktop is installing podman itself.

[Luap99]

full layers are cached but not if you interrupt during the layer download: #14616

The version in the machine is not updated automatically, you would need something like podman machine os apply quay.io/podman/machine-os:5.2 to update the VM image.

[finch71]

users may interrupt when internet is slow and restart when it's faster.
I think #14616 worth reconsideration