K3s agent won't start on Linux kernel 6.1.54 (and latest Raspberry Pi OS)

[basphilippus]

Environmental Info:
K3s Version:
k3s version v1.28.3+k3s2 (bbafb86)
go version go1.20.10

Node(s) CPU architecture, OS, and Version:
Linux pi-node-2 6.1.0-rpi4-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.54-1+rpt2 (2023-10-05) aarch64 GNU/Linux

Cluster Configuration:
1 server, 3 agents

Describe the bug:
The released version of Raspberry Pi OS is now on kernel 6.1.0-rpi4-rpi-2712 (which is Linux kernel 6.1.54), which drops support for /sys/fs/cgroup/memory/memory.kmem.limit_in_bytes. (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/Documentation/admin-guide/cgroup-v1/memory.rst?h=linux-6.1.y&id=21ef9e11205fca43785eecf7d4a99528d4de5701)
K3s (or possibly all of Kubernetes) requires kmem.limit_in_bytes to start and won't anymore.

Steps To Reproduce:

• Get a Linux version with kernel 6.1.54 or Raspberry Pi OS with kernel 6.1.0-rpi4-rpi-2712
• Install K3s (any version)
• The agent (and possibly the server as well) won't start because /sys/fs/cgroup/memory/memory.kmem.limit_in_bytes can't be found.

Expected behavior:
K3s should possibly not need memory.kmem.limit_in_byte to start up anymore.

Actual behavior:
K3s startup fails because /sys/fs/cgroup/memory/memory.kmem.limit_in_bytes can't be found.

Additional context / logs:

Nov 23 22:18:55 pi-node-2 k3s[7015]: I1123 22:18:55.258722 7015 kubelet.go:2303] "Starting kubelet main sync loop"
Nov 23 22:18:55 pi-node-2 k3s[7015]: E1123 22:18:55.258779 7015 kubelet.go:2327] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 23 22:18:55 pi-node-2 k3s[7015]: W1123 22:18:55.272449 7015 container.go:586] Failed to update stats for container "/": openat2 /sys/fs/cgroup/memory/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 23 22:18:55 pi-node-2 k3s[7015]: W1123 22:18:55.272776 7015 container.go:586] Failed to update stats for container "/system.slice/k3s-agent.service": openat2 /sys/fs/cgroup/memory/system.slice/k3s-agent.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 23 22:18:55 pi-node-2 k3s[7015]: time="2023-11-23T22:18:55+01:00" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=pi-node-2 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Nov 23 22:18:55 pi-node-2 k3s[7015]: E1123 22:18:55.288477 7015 container_manager_linux.go:881] "Unable to get rootfs data from cAdvisor interface" err="unable to find data in memory cache"
Nov 23 22:18:55 pi-node-2 k3s[7015]: I1123 22:18:55.289526 7015 cpu_manager.go:214] "Starting CPU manager" policy="none"
Nov 23 22:18:55 pi-node-2 k3s[7015]: I1123 22:18:55.289544 7015 cpu_manager.go:215] "Reconciling" reconcilePeriod="10s"
Nov 23 22:18:55 pi-node-2 k3s[7015]: I1123 22:18:55.289562 7015 state_mem.go:36] "Initialized new in-memory state store"
Nov 23 22:18:55 pi-node-2 k3s[7015]: I1123 22:18:55.289724 7015 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Nov 23 22:18:55 pi-node-2 k3s[7015]: I1123 22:18:55.289748 7015 state_mem.go:96] "Updated CPUSet assignments" assignments={}
Nov 23 22:18:55 pi-node-2 k3s[7015]: I1123 22:18:55.289757 7015 policy_none.go:49] "None policy: Start"
Nov 23 22:18:55 pi-node-2 k3s[7015]: I1123 22:18:55.290192 7015 memory_manager.go:169] "Starting memorymanager" policy="None"
Nov 23 22:18:55 pi-node-2 k3s[7015]: I1123 22:18:55.290331 7015 state_mem.go:35] "Initializing new in-memory state store"
Nov 23 22:18:55 pi-node-2 k3s[7015]: I1123 22:18:55.290624 7015 state_mem.go:75] "Updated machine memory state"
Nov 23 22:18:55 pi-node-2 k3s[7015]: E1123 22:18:55.290736 7015 kubelet.go:1511] "Failed to start ContainerManager" err="failed to get rootfs info: unable to find data in memory cache"
Nov 23 22:18:55 pi-node-2 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=1/FAILURE

Please let me know if this bug report belongs with K3s or in the Kubernetes repository.

[basphilippus]

Linux kernel 6.1.55 revert the dropping of /sys/fs/cgroup/memory/memory.kmem.limit_in_bytes, but I do believe that /sys/fs/cgroup/memory/memory.kmem.limit_in_bytes is deprecated and will be removed permanently at some point.

[brandond]

Closing as upstream issue with kubelet + distro kernel packaging.

xref to the other issue you opened: RPi-Distro/repo#352