Failed to start on Fedora 41, rootless

[metal3d]

Environmental Info:
K3s Version: v1.31.2+k3s1

Node(s) CPU architecture, OS, and Version: amd64, Fedora Linux 41

Cluster Configuration: default

Describe the bug:

I only tried to start the server the "easy way" :

server --rootless --selinux
INFO[0000] Starting k3s v1.31.2+k3s1 (6da20424)         
INFO[0000] Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s 
INFO[0000] Configuring database table schema and indexes, this may take a moment... 
INFO[0000] Database tables and indexes are up to date   
INFO[0000] Kine available at unix://kine.sock           
INFO[0000] Reconciling bootstrap data between datastore and disk 
INFO[0000] Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/home/metal3d/.rancher/k3s/server/tls/temporary-certs --client-ca-file=/home/metal3d/.rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/home/metal3d/.rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --etcd-servers=unix://kine.sock --kubelet-certificate-authority=/home/metal3d/.rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/home/metal3d/.rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/home/metal3d/.rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/home/metal3d/.rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/home/metal3d/.rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/home/metal3d/.rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/home/metal3d/.rancher/k3s/server/tls/service.key --service-account-signing-key-file=/home/metal3d/.rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/home/metal3d/.rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/home/metal3d/.rancher/k3s/server/tls/serving-kube-apiserver.key 
INFO[0000] Running kube-scheduler --authentication-kubeconfig=/home/metal3d/.rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/home/metal3d/.rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/home/metal3d/.rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259 
W1122 13:00:01.681370      57 registry.go:256] calling componentGlobalsRegistry.AddFlags more than once, the registry will be set by the latest flags
INFO[0000] Waiting for API server to become available   
INFO[0000] Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/home/metal3d/.rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/home/metal3d/.rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/home/metal3d/.rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/home/metal3d/.rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/home/metal3d/.rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/home/metal3d/.rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/home/metal3d/.rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/home/metal3d/.rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/home/metal3d/.rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/home/metal3d/.rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --kubeconfig=/home/metal3d/.rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/home/metal3d/.rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/home/metal3d/.rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true 
W1122 13:00:01.681787      57 registry.go:256] calling componentGlobalsRegistry.AddFlags more than once, the registry will be set by the latest flags
I1122 13:00:01.681922      57 options.go:228] external host was not specified, using 10.41.0.100
INFO[0000] Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/home/metal3d/.rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/home/metal3d/.rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/home/metal3d/.rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --feature-gates=CloudDualStackNodeIPs=true --kubeconfig=/home/metal3d/.rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false 
INFO[0000] Server node token is available at /home/metal3d/.rancher/k3s/server/token 
INFO[0000] To join server node to cluster: k3s server -s https://10.41.0.100:6443 -t ${SERVER_NODE_TOKEN} 
INFO[0000] Agent node token is available at /home/metal3d/.rancher/k3s/server/agent-token 
INFO[0000] To join agent node to cluster: k3s agent -s https://10.41.0.100:6443 -t ${AGENT_NODE_TOKEN} 
I1122 13:00:01.682938      57 server.go:150] Version: v1.31.2+k3s1
I1122 13:00:01.682972      57 server.go:152] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
INFO[0000] Wrote kubeconfig /home/metal3d/.kube/k3s.yaml 
INFO[0000] Run: k3s kubectl                             
FATA[0000] failed to find cpuset cgroup (v2)            
FATA[0000] child died: command [k3s server --rootless --selinux] exited: exit status 1 
FATA[0000] child exited: exit status 1                  

We're using cgroup v2, and the groups exists:

cat /proc/cgroups 
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	483	1
cpu	0	483	1
cpuacct	0	483	1
blkio	0	483	1
memory	0	483	1
devices	0	483	1
freezer	0	483	1
net_cls	0	483	1
perf_event	0	483	1
net_prio	0	483	1
hugetlb	0	483	1
pids	0	483	1
rdma	0	483	1
misc	0	483	1

I tried to add cpuset and cgroup in:

$ cat /etc/systemd/system/[email protected]/delegate.conf 
[Service]
Delegate=cpu cpuset io memory pids

$ sudo systemctl daemon-reload

$ cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers
cpuset cpu io memory pids

And daemon-reload afterward. The error remains.

[metal3d]

OK... after having changed the delegation, I needed to reboot.