Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: database graph driver "" does not match our graph driver "overlay": database configuration mismatch #24738

Open
shenki opened this issue Dec 2, 2024 · 0 comments · May be fixed by #24775
Open

Error: database graph driver "" does not match our graph driver "overlay": database configuration mismatch #24738

shenki opened this issue Dec 2, 2024 · 0 comments · May be fixed by #24775
Assignees
@mheon
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@shenki
Copy link

shenki commented Dec 2, 2024

Issue Description

Since November, a server has been logging this error when attempting an auto-update cronjob:

podman[895410]: Error: database graph driver "" does not match our graph driver "overlay": database configuration mismatch

During this time period my distro upgraded from 5.2.1 to 5.2.2.

I am now running 5.2.5.

Steps to reproduce the issue

Since November, a server has been logging an error when attempting an auto-update cronjob.

Steps to reproduce the issue

  1. Use podman with auto-update cronjob

Describe the results you received

podman[895410]: Error: database graph driver "" does not match our graph driver "overlay": database configuration mismatch

During this time period my distro upgraded from 5.2.1 to 5.2.2.

I am now running 5.2.5.

Describe the results you expected

I expected the cronjob to succeed

podman info output

$ podman info
host:
  arch: arm64
  buildahVersion: 1.37.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.12-3_arm64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 99.64
    systemPercent: 0.17
    userPercent: 0.19
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: trixie
    distribution: debian
    version: unknown
  eventLogger: journald
  freeLocks: 2036
  hostname: hubble
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.11.0
  linkmode: dynamic
  logDriver: journald
  memFree: 3758686208
  memTotal: 16489906176
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.12.2-1_arm64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark_1.12.1-3_arm64
    path: /usr/lib/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: runc
    package: runc_1.1.15+ds1-1_arm64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.15+ds1
      commit: 1.1.15+ds1-1
      spec: 1.2.0
      go: go1.23.2
      libseccomp: 2.5.5
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20241030.ee7d0b6-1_arm64
    version: |
      pasta 0.0~git20241030.ee7d0b6-1
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1+b2_arm64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 1477h 41m 29.00s (Approximately 61.54 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/joel/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 7
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/joel/.local/share/containers/storage
  graphRootAllocated: 111714734080
  graphRootUsed: 105411731456
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 7
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/joel/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.5
  Built: 1729890366
  BuiltTime: Sat Oct 26 07:36:06 2024
  GitCommit: ""
  GoVersion: go1.23.2
  Os: linux
  OsArch: linux/arm64
  Version: 5.2.5

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Debian testing

Additional information

I upgraded to the latest version 5.3.1. The error is now:

Dec 03 09:26:41 hubble systemd[1]: Starting podman-restart.service - Podman Start All Containers With Restart Policy Set To Always...
Dec 03 09:26:41 hubble podman[959599]: time="2024-12-03T09:26:41+10:30" level=info msg="/usr/bin/podman filtering at log level info"
Dec 03 09:26:41 hubble podman[959599]: time="2024-12-03T09:26:41+10:30" level=info msg="Using sqlite as database backend"
Dec 03 09:26:41 hubble podman[959599]: Error: database graph driver "" does not match our graph driver "overlay": database configuration mismatch
Dec 03 09:26:41 hubble systemd[1]: podman-restart.service: Main process exited, code=exited, status=125/n/a

The top hit on DDG for this issue is #7396, which was closed.
@shenki shenki added the kind/bug Categorizes issue or PR as related to a bug. label Dec 2, 2024
@mheon mheon self-assigned this Dec 5, 2024
mheon added a commit to mheon/libpod that referenced this issue Dec 5, 2024
@mheon
In SQLite state, use defaults for empty-string checks
5ab154f 
As part of our database init, we perform a check of the current
values for a few fields (graph driver, graph root, static dir,
and a few more) to validate that Libpod is being started with a
sane & sensible config, and the user's containers can actually be
expected to work. Basically, we take the current runtime config
and compare against values cached in the database from the first
time Podman was run.

We've had some issues with this logic before this year around
symlink resolution, but this is a new edge case. Somehow, the
database is being loaded with the empty string for some fields
(at least graph driver) which is causing comparisons to fail
because we will never compare against "" for those fields - we
insert the default value instead, assuming we have one.

Having a value of "" in the database largely invalidates the
check so arguably we could just drop it, but what BoltDB did -
and what SQLite does after this patch - is to use the default
value for comparison instead of "". This should still catch some
edge cases, and shouldn't be too harmful.

What this does not do is identify or solve the reason that we are
seeing the empty string in the database at all. From my read on
the logic, it must mean that the graph driver is explicitly set
to "" in the c/storage config at the time Podman is first run and
I'm not precisely sure how that happens.

Fixes containers#24738

Signed-off-by: Matt Heon <[email protected]>
@mheon mheon linked a pull request Dec 5, 2024 that will close this issue
Open
mheon added a commit to mheon/libpod that referenced this issue Dec 10, 2024
@mheon
In SQLite state, use defaults for empty-string checks
7df556c 
As part of our database init, we perform a check of the current
values for a few fields (graph driver, graph root, static dir,
and a few more) to validate that Libpod is being started with a
sane & sensible config, and the user's containers can actually be
expected to work. Basically, we take the current runtime config
and compare against values cached in the database from the first
time Podman was run.

We've had some issues with this logic before this year around
symlink resolution, but this is a new edge case. Somehow, the
database is being loaded with the empty string for some fields
(at least graph driver) which is causing comparisons to fail
because we will never compare against "" for those fields - we
insert the default value instead, assuming we have one.

Having a value of "" in the database largely invalidates the
check so arguably we could just drop it, but what BoltDB did -
and what SQLite does after this patch - is to use the default
value for comparison instead of "". This should still catch some
edge cases, and shouldn't be too harmful.

What this does not do is identify or solve the reason that we are
seeing the empty string in the database at all. From my read on
the logic, it must mean that the graph driver is explicitly set
to "" in the c/storage config at the time Podman is first run and
I'm not precisely sure how that happens.

Fixes containers#24738

Signed-off-by: Matt Heon <[email protected]>
mheon added a commit to mheon/libpod that referenced this issue Dec 10, 2024
@mheon
In SQLite state, use defaults for empty-string checks
1e04b52 
As part of our database init, we perform a check of the current
values for a few fields (graph driver, graph root, static dir,
and a few more) to validate that Libpod is being started with a
sane & sensible config, and the user's containers can actually be
expected to work. Basically, we take the current runtime config
and compare against values cached in the database from the first
time Podman was run.

We've had some issues with this logic before this year around
symlink resolution, but this is a new edge case. Somehow, the
database is being loaded with the empty string for some fields
(at least graph driver) which is causing comparisons to fail
because we will never compare against "" for those fields - we
insert the default value instead, assuming we have one.

Having a value of "" in the database largely invalidates the
check so arguably we could just drop it, but what BoltDB did -
and what SQLite does after this patch - is to use the default
value for comparison instead of "". This should still catch some
edge cases, and shouldn't be too harmful.

What this does not do is identify or solve the reason that we are
seeing the empty string in the database at all. From my read on
the logic, it must mean that the graph driver is explicitly set
to "" in the c/storage config at the time Podman is first run and
I'm not precisely sure how that happens.

Fixes containers#24738

Signed-off-by: Matt Heon <[email protected]>
mheon added a commit to mheon/libpod that referenced this issue Dec 10, 2024
@mheon
In SQLite state, use defaults for empty-string checks
b54620d 
As part of our database init, we perform a check of the current
values for a few fields (graph driver, graph root, static dir,
and a few more) to validate that Libpod is being started with a
sane & sensible config, and the user's containers can actually be
expected to work. Basically, we take the current runtime config
and compare against values cached in the database from the first
time Podman was run.

We've had some issues with this logic before this year around
symlink resolution, but this is a new edge case. Somehow, the
database is being loaded with the empty string for some fields
(at least graph driver) which is causing comparisons to fail
because we will never compare against "" for those fields - we
insert the default value instead, assuming we have one.

Having a value of "" in the database largely invalidates the
check so arguably we could just drop it, but what BoltDB did -
and what SQLite does after this patch - is to use the default
value for comparison instead of "". This should still catch some
edge cases, and shouldn't be too harmful.

What this does not do is identify or solve the reason that we are
seeing the empty string in the database at all. From my read on
the logic, it must mean that the graph driver is explicitly set
to "" in the c/storage config at the time Podman is first run and
I'm not precisely sure how that happens.

Fixes containers#24738

Signed-off-by: Matt Heon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mheon mheon

Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

In SQLite state, use defaults for empty-string checks mheon/libpod
2 participants
@shenki @mheon