catastrophic failure if data deleted from .local/share/containers/storage/overlay/

[cob-web-corner]

Issue Description

A disk failed that was mapped to .local/share/containers/storage/overlay/ and the disk has since been replaced.

Unfortunately due to the missing data just about everything is failing. Funnily enough the containers still managed to run but running podman exec resulting in path not found errors

Running system prune -a results in

Error: determining size of layer with ID "<ID>": faccessat /home/duser/.local/share/containers/storage/overlay/<ID>: no such file or directory

same with any other system cleanup command

Steps to reproduce the issue

Steps to reproduce the issue

1. Run a couple containers
2. Delete anything under .local/share/containers/storage/overlay/
3. Try to run any cleanup commands to fix the situation

Describe the results you received

Error: determining size of layer with ID "<ID>": faccessat /home/duser/.local/share/containers/storage/overlay/<ID>: no such file or directory

Describe the results you expected

System would be able to determine recover from a position like this able to remove orphan entries in it's database and json files

podman info output

host:
  arch: amd64
  buildahVersion: 1.37.1
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.1.12, commit: 3bc422cd8aaec542d85d1a80f2d38e6e69046b5b'
  cpuUtilization:
    idlePercent: 99.74
    systemPercent: 0.15
    userPercent: 0.12
  cpus: 4
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: file
  freeLocks: 2025
  hostname: mka0732
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.8.0-45-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 7419215872
  memTotal: 8226447360
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.16.1
      commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20240220.1e6f92b-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 44m 40.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  '*':
    Blocked: false
    Insecure: true
    Location: nexus.mkalacim.osi
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: '*'
    PullFromMirror: ""
  search:
  - nexus.mkalacim.osi
store:
  configFile: /home/duser/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 0
    stopped: 6
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/duser/.local/share/containers/storage
  graphRootAllocated: 40483942400
  graphRootUsed: 16493948928
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 6
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /home/duser/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.1
  Built: 1724060675
  BuiltTime: Mon Aug 19 05:44:35 2024
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[giuseppe]

system prune is not meant to be used with a corrupted storage.

There are already two other commands to help in this case: system check -r and system reset.

[cob-web-corner]

This is a RTFM moment. Thank you!

In a panic and being new to podman I ended up doing system reset manually but its great to know that command exists.