Rootless K3s fails with "operation not permitted" error during mount operations

[nishantmunjal7]

When running K3s in rootless mode, I'm encountering an "operation not permitted" error during mount operations. This appears to be related to the handling of mount flags in user namespaces.

Environmental Info:
K3s Version:
k3s version v1.30.4+k3s1 (98262b5)
go version go1.22.5

Node(s) CPU architecture, OS, and Version:
Linux ip-** 5.14.0-427.31.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 9 14:06:03 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug:
failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting ".rancher/k3s/agent/kubelet/pods//volumes/kubernetes.io ~ configmap/config-volume" to rootfs at "/etc/coredns": mount .rancher/k3s/agent/kubelet/pods//volumes/kubernetes.io~configmap/config-volume:/etc/coredns (via /proc/self/fd/6), flags: 0x5021: operation not permitted: unknown" containerID=""

Steps To Reproduce:

• 1. Set up K3s in rootless mode
• 2. Create a test directory and mount it with specific options

mkdir -p ~/test_dir
sudo mount -o bind,rw,nosuid,nodev ~/somedir /home/user/.rancher

• 3. Attempt to start K3s Rootless service from user folder.

Expected behavior:
All K3s pods should start and operate normally in rootless mode.

Actual behavior:
All pods are in CrashLoopBackOff State

Additional context / logs:
I noticed a similar issue for this in Moby repo, do we need to implement something similar with K3s containerd version

[brandond]

sudo mount -o bind,rw,nosuid,nodev ~/somedir /home/user/.rancher

Why are you bind-mounting the rootless dir with weird options? Why not just set --data-dir=$HOME/somedir instead?