panic: invalid page type: 9: 10

[arogozhnikov]

Issue Description

Can't podman run hello-world on a new instance, but can sudo podman run hello-world.

I have a similar setup on other set of instances, but I can't figure what's the difference between the two.

I've changed configuration to use other drive (nvme) with storage.conf and moved storages to local nvme, because home folder is a mounted NFS.

Things I've tried (no success in all cases, and I receive the same error posted below):

• enabling/disabling lingering
• reinstalling podman
• changing locations to /test, which is part of root volume
• deleting storage location (this one is likely implicated, as on some of instances under identical setup this worked, but after deleting storage and bolts.db all nodes show the same error)

Steps to reproduce the issue

(see above)

Describe the results you received

Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull quay.io/podman/hello:latest...
Getting image source signatures
panic: invalid page type: 9: 10

goroutine 346 [running]:
github.com/containers/podman/vendor/go.etcd.io/bbolt.(*Cursor).search(0xc000ce3470, {0x5e8b8574aa50, 0x12, 0x12}, 0x5e8b83a3fb8f?)
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/cursor.go:274 +0x28b
github.com/containers/podman/vendor/go.etcd.io/bbolt.(*Cursor).seek(0xc000ce3470, {0x5e8b8574aa50?, 0x78030850af60?, 0x0?})
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/cursor.go:159 +0x36
github.com/containers/podman/vendor/go.etcd.io/bbolt.(*Bucket).Bucket(0xc000aec558, {0x5e8b8574aa50, 0x12, 0x12})
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/bucket.go:105 +0xd8
github.com/containers/podman/vendor/go.etcd.io/bbolt.(*Tx).Bucket(...)
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/tx.go:102
github.com/containers/podman/vendor/github.com/containers/image/v5/pkg/blobinfocache/boltdb.(*cache).uncompressedDigest(0xc000b50d00?, 0xc000b50d20?, {0xc000dd44b0, 0x47})
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/github.com/containers/image/v5/pkg/blobinfocache/boltdb/boltdb.go:153 +0x67
github.com/containers/podman/vendor/github.com/containers/image/v5/pkg/blobinfocache/boltdb.(*cache).UncompressedDigest.func1(0x44?)
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/github.com/containers/image/v5/pkg/blobinfocache/boltdb/boltdb.go:182 +0x34
github.com/containers/podman/vendor/go.etcd.io/bbolt.(*DB).View(0xc000dd40f0?, 0xc000ce3718)
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:893 +0x82
github.com/containers/podman/vendor/github.com/containers/image/v5/pkg/blobinfocache/boltdb.(*cache).view(0xc0003d66b0, 0xc000a49f70?)
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/github.com/containers/image/v5/pkg/blobinfocache/boltdb/boltdb.go:131 +0x1b5
github.com/containers/podman/vendor/github.com/containers/image/v5/pkg/blobinfocache/boltdb.(*cache).UncompressedDigest(0x0?, {0xc000dd44b0?, 0xc000d00280?})
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/github.com/containers/image/v5/pkg/blobinfocache/boltdb/boltdb.go:181 +0x65
github.com/containers/podman/vendor/github.com/containers/image/v5/copy.(*imageCopier).copyLayer(0xc000bd24b0, {0x5e8b84b62628, 0xc000ac9cb0}, {{0xc000dd44b0, 0x47}, 0x532a9, {0x0, 0x0, 0x0}, 0x0, ...}, ...)
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/github.com/containers/image/v5/copy/single.go:606 +0x2b7
github.com/containers/podman/vendor/github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1(0x0, {{0xc000dd44b0, 0x47}, 0x532a9, {0x0, 0x0, 0x0}, 0x0, {0xc000780fc0, 0x31}, ...}, ...)
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/github.com/containers/image/v5/copy/single.go:410 +0x3dd
created by github.com/containers/podman/vendor/github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2
	/home/abuild/debbuild/BUILD/podman-4.6.2/_build/src/github.com/containers/podman/vendor/github.com/containers/image/v5/copy/single.go:447 +0x37e

Describe the results you expected

container normally started.

podman info output

ost:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.12-0ubuntu22.04+obs20.12_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 99.55
    systemPercent: 0.15
    userPercent: 0.3
  cpus: 255
  databaseBackend: boltdb
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  freeLocks: 2048
  hostname: compute-permanent-node-973
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.5.0-1025-oracle
  linkmode: dynamic
  logDriver: journald
  memFree: 2125290926080
  memTotal: 2164235874304
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.6.0
    package: netavark_1.3.0-0ubuntu22.04+obs22.9_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.3.0
  ociRuntime:
    name: crun
    package: crun_101:1.14.4-0ubuntu22.04+obs70.24_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.0.1-2_amd64
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 0
  swapTotal: 0
  uptime: 2h 20m 38.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/ubuntu/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /chaitest/.podman/graphroot
  graphRootAllocated: 532608356352
  graphRootUsed: 21829300224
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /chaitest/.podman/runroot
  transientStore: false
  volumePath: /chaitest/.podman/graphroot/volumes
version:
  APIVersion: 4.6.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

ubuntu22, podman was installed from kubic (I have another setup with ~identical parameters, and it works)

Additional information

No response

[Luap99]

your boltdb database is corrupted (did you have a system crash?). You have to remove the boltdb file (~/.local/share/containers/storage/libpod/bolt_state.db by default) then it will work again, of course you loose everything that was stored there, containers, pods, etc... If you have critical files backup them first.

Newer podman versions use sqlite by default.

[arogozhnikov]

as I mentioned, I've deleted boltdb several times, but problem persists. Ok, I'll try a new podman

[Luap99]

Can you run with --log-level debug, did you actually deleted the paths listed theret? It is not clear to me how such error would be reproducible.

You might want to check issues on https://github.com/etcd-io/bbolt

But as I said as of 4.8 new installations will default to sqlite so this should not be a problem there, old installes will still keep using boltdb until you remove the database (i.e podman system reset)