OCI runtime error: unable to start container: crun: cannot set memory swappiness with cgroupv2

[Bush-cat]

Issue Description

To update my Containers I recreated them, but after they were created, most couldn't start anymore. 4 out of 7 of my Containers got that Error.
I guess the other Containers do not have a swappiness flag. Or is it possible to globally disable only swapiness since cgroupv2 doesn't allow it. Or should I use cgroupv1 instead, is it posible to set cgroupversion per Container? I tried podman updatebutit requiresthe container to be running.
Everything worked before, I'm not sure if podman used cgroup v2 before or it changed with podman v5 or decided to use it with a kernel update

Steps to reproduce the issue

Steps to reproduce the issue

1. podman pull
2. podman stop
3. podman rm
4. podman run

Describe the results you received

Error: OCI runtime error: unable to start container "1b5525031c6f817feb4009377f0f032ce8e0940c8bc5b7382631dc9a7ffe1340": crun: cannot set memory swappiness with cgroupv2

Describe the results you expected

Container starting Successfully

podman info output

host:
  arch: amd64
  buildahVersion: 1.35.3
  cgroupControllers:
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.11-1.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 99.23
    systemPercent: 0.32
    userPercent: 0.45
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: opensuse-microos
    version: "20240512"
  eventLogger: journald
  freeLocks: 2024
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.8-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 15588700160
  memTotal: 16771973120
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.3.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.2.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.2.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-20240426.d03c4e2-2.1.x86_64
    version: |
      pasta 20240426.d03c4e2-2.1
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.0-1.1.x86_64
    version: |-
      slirp4netns version 1.3.0
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 45m 27.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.opensuse.org
  - registry.suse.com
store:
  configFile: /home/bcat/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 3
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/bcat/.local/share/containers/storage
  graphRootAllocated: 34335600640
  graphRootUsed: 11060162560
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 15
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/bcat/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.2
  Built: 1713529050
  BuiltTime: Fri Apr 19 14:17:30 2024
  GitCommit: ""
  GoVersion: go1.21.9
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

OpenSuse MicroOS Podman Container Host Image running as VM in Proxmox

Additional information

No response

[Luap99]

cgroupv2 doesn't allow you to set swappiness so I suggest you simply remove that setting. The cgroup version is defined globally and depens on the system configuration.

I think you can still configure swappiness globally via sysctl.

[Bush-cat]

I didn't configure swappiness, yesterday I recreated the containers again and they are working now, I'm not sure how they got the swappiness configuration

[rhatdan]

Did you create the containers with an older podman and then update or switch to cgoupv2?

[Bush-cat]

possible, I was running podman v4 before, I'm not sure when I updated podman since it happens automatically in opensuse microos

[chisaato]

Hey, here are some new research abount podman and swappiness problems

containrrr/watchtower#1060

I think podman should also report MemorySwappiness to null if it is not set.