[Release-1.31] - K3s extracts to wrong dir when embedded subcommands (kubectl, ctr, crictl) are run with --help or --version flags

[brandond]

Backport fix for K3s extracts to wrong dir when embedded subcommands (kubectl, ctr, crictl) are run with --help or --version flags

• K3s extracts to wrong dir when embedded subcommands (kubectl, ctr, crictl) are run with --help or --version flags #11235

[aganesh-suse]

Validated on release-1.31 branch with commit 4a3ea1c

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server
data-dir: "/home/ubuntu/.rancher/k3s"

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_COMMIT='4a3ea1c3ac909b4361da9b75cdc4af4f12525803' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

4. Verify help/version outputs for ctr, crictl and kubectl - using regular/sudo user, and using/not using full path to binary.

$ K3S_DEBUG=true k3s ctr --help 
$ K3S_DEBUG=true k3s ctr --version
$ K3S_DEBUG=true sudo k3s ctr --help 
$ K3S_DEBUG=true sudo k3s ctr --version

$ K3S_DEBUG=true k3s crictl --help 
$ K3S_DEBUG=true k3s crictl --version
$ K3S_DEBUG=true sudo k3s crictl --help 
$ K3S_DEBUG=true sudo k3s crictl --version

$ K3S_DEBUG=true k3s kubectl --help 
$ K3S_DEBUG=true k3s kubectl --version
$ K3S_DEBUG=true sudo k3s kubectl --help 
$ K3S_DEBUG=true sudo k3s kubectl --version

$ kubectl --help
$ kubectl --version
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml version
$ K3S_DEBUG=true sudo /usr/local/bin/k3s ctr --help 
$ K3S_DEBUG=true sudo /usr/local/bin/k3s ctr --version
$ /usr/bin/ctr --version
$ /usr/local/bin/crictl --version
$ ls -lrt | grep help
$ ls -lrt | grep version

Replication Results:

• k3s version used for replication:

$ k3s -v
k3s version v1.31.2+k3s1 (6da20424)
go version go1.22.8

Version results:

$ K3S_DEBUG=true  k3s ctr --version 
time="2024-11-13T20:27:11Z" level=debug msg="Asset dir /home/ubuntu/--version/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:27:11Z" level=debug msg="Running /home/ubuntu/--version/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/ctr [ctr --version]"
ctr github.com/k3s-io/containerd v1.7.22-k3s1 

$ K3S_DEBUG=true  k3s crictl --version 
time="2024-11-13T20:27:14Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
time="2024-11-13T20:27:14Z" level=debug msg="Asset dir /home/ubuntu/--version/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:27:14Z" level=debug msg="Running /home/ubuntu/--version/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/crictl [crictl --version]"
crictl version v1.31.0-k3s2

$ K3S_DEBUG=true  k3s kubectl version 
time="2024-11-13T20:27:16Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:27:16Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/kubectl [kubectl version]"
Client Version: v1.31.2+k3s1
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s1

$ kubectl version 
time="2024-11-13T20:27:19Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:27:19Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/kubectl [kubectl version]"
Client Version: v1.31.2+k3s1
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s1

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml version 
time="2024-11-13T20:27:21Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:27:21Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/kubectl [kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml version]"
Client Version: v1.31.2+k3s1
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s1

$  /usr/bin/ctr --version 
ctr github.com/containerd/containerd 1.7.12

$  /usr/local/bin/crictl --version 
time="2024-11-13T20:27:26Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
crictl version v1.31.0-k3s2

$ K3S_DEBUG=true  /usr/local/bin/k3s ctr --version 
time="2024-11-13T20:27:29Z" level=debug msg="Asset dir /home/ubuntu/--version/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:27:29Z" level=debug msg="Running /home/ubuntu/--version/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/ctr [ctr --version]"
ctr github.com/k3s-io/containerd v1.7.22-k3s1 

$ K3S_DEBUG=true sudo k3s ctr --version 
ctr github.com/k3s-io/containerd v1.7.22-k3s1 

$ K3S_DEBUG=true sudo k3s crictl --version 
crictl version v1.31.0-k3s2

$ K3S_DEBUG=true sudo k3s kubectl version 
time="2024-11-13T20:27:39Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:27:39Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/kubectl [kubectl version]"
Client Version: v1.31.2+k3s1
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s1

$ kubectl version 
time="2024-11-13T20:27:41Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:27:41Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/kubectl [kubectl version]"
Client Version: v1.31.2+k3s1
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s1

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml version 
time="2024-11-13T20:27:44Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:27:44Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/kubectl [kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml version]"
Client Version: v1.31.2+k3s1
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s1

$ sudo /usr/bin/ctr --version 
ctr github.com/containerd/containerd 1.7.12

$ sudo /usr/local/bin/crictl --version 
crictl version v1.31.0-k3s2

$ K3S_DEBUG=true sudo /usr/local/bin/k3s ctr --version 
ctr github.com/k3s-io/containerd v1.7.22-k3s1 

Help results:

$ K3S_DEBUG=true  k3s ctr --help 
time="2024-11-13T18:47:13Z" level=debug msg="Asset dir /home/ubuntu/--help/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T18:47:13Z" level=debug msg="Running /home/ubuntu/--help/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/ctr [ctr --help]"

$ K3S_DEBUG=true  k3s crictl --help 
time="2024-11-13T18:47:18Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
time="2024-11-13T18:47:18Z" level=debug msg="Asset dir /home/ubuntu/--help/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T18:47:18Z" level=debug msg="Running /home/ubuntu/--help/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/crictl [crictl --help]"

$ K3S_DEBUG=true  k3s kubectl help 
time="2024-11-13T18:47:24Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T18:47:24Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/kubectl [kubectl help]"

$ kubectl help 
time="2024-11-13T18:47:29Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T18:47:29Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/kubectl [kubectl help]"

$ K3S_DEBUG=true  /usr/local/bin/k3s ctr --help 
time="2024-11-13T20:30:59Z" level=debug msg="Asset dir /home/ubuntu/--help/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:30:59Z" level=debug msg="Running /home/ubuntu/--help/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/ctr [ctr --help]"

$ K3S_DEBUG=true sudo k3s kubectl help 
time="2024-11-13T20:31:22Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1"
time="2024-11-13T20:31:22Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/3345fdb78d4ac6f55d7d70b8ec401ed32d58d5af6b2e11412cd5a2d3c50ff3d1/bin/kubectl [kubectl help]"

$ ls -lrt /home/ubuntu | grep version
drwxr-xr-x 3 ubuntu ubuntu  4096 Nov 13 17:45 --version
$ ls -lrt /home/ubuntu | grep help
drwxr-xr-x 3 ubuntu ubuntu  4096 Nov 13 17:48 --help

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.31.2+k3s-4a3ea1c3 (4a3ea1c3)
go version go1.22.8

======= Execute START ========

$ K3S_DEBUG=true  k3s ctr --version 
time="2024-11-13T20:28:02Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:28:02Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/ctr [ctr --version]"
ctr github.com/k3s-io/containerd v1.7.23-k3s1 

$ K3S_DEBUG=true  k3s crictl --version 
time="2024-11-13T20:28:08Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
time="2024-11-13T20:28:08Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
time="2024-11-13T20:28:08Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:28:08Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/crictl [crictl --version]"
crictl version v1.31.0-k3s2

$ K3S_DEBUG=true  k3s kubectl version 
time="2024-11-13T20:28:14Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:28:14Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl version]"
Client Version: v1.31.2+k3s-4a3ea1c3
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s-4a3ea1c3

$ kubectl version 
time="2024-11-13T20:28:19Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:28:19Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl version]"
Client Version: v1.31.2+k3s-4a3ea1c3
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s-4a3ea1c3

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml version 
time="2024-11-13T20:28:25Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:28:25Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml version]"
Client Version: v1.31.2+k3s-4a3ea1c3
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s-4a3ea1c3

$  /usr/bin/ctr --version 
ctr github.com/containerd/containerd 1.7.12

$  /usr/local/bin/crictl --version 
time="2024-11-13T20:28:36Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
time="2024-11-13T20:28:36Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
time="2024-11-13T20:28:36Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:28:36Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/crictl [crictl --version]"
crictl version v1.31.0-k3s2

$ K3S_DEBUG=true  /usr/local/bin/k3s ctr --version 
time="2024-11-13T20:28:41Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:28:41Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/ctr [ctr --version]"
ctr github.com/k3s-io/containerd v1.7.23-k3s1 

$ K3S_DEBUG=true sudo k3s ctr --version 
time="2024-11-13T20:28:52Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:28:52Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/ctr [ctr --version]"
ctr github.com/k3s-io/containerd v1.7.23-k3s1 

$ K3S_DEBUG=true sudo k3s crictl --version 
time="2024-11-13T20:28:57Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:28:57Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/crictl [crictl --version]"
crictl version v1.31.0-k3s2

$ K3S_DEBUG=true sudo k3s kubectl version 
time="2024-11-13T20:29:03Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:29:03Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl version]"
Client Version: v1.31.2+k3s-4a3ea1c3
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s-4a3ea1c3

$ kubectl version 
time="2024-11-13T20:29:08Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:29:08Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl version]"
Client Version: v1.31.2+k3s-4a3ea1c3
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s-4a3ea1c3

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml version 
time="2024-11-13T20:29:15Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:29:15Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml version]"
Client Version: v1.31.2+k3s-4a3ea1c3
Kustomize Version: v5.4.2
Server Version: v1.31.2+k3s-4a3ea1c3

$ sudo /usr/bin/ctr --version 
ctr github.com/containerd/containerd 1.7.12

$ sudo /usr/local/bin/crictl --version 
time="2024-11-13T20:29:25Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:29:25Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/crictl [crictl --version]"
crictl version v1.31.0-k3s2

$ K3S_DEBUG=true sudo /usr/local/bin/k3s ctr --version 
time="2024-11-13T20:29:31Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:29:31Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/ctr [ctr --version]"
ctr github.com/k3s-io/containerd v1.7.23-k3s1 

Help results:

$ K3S_DEBUG=true  k3s ctr --help 
time="2024-11-13T20:32:00Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:32:00Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/ctr [ctr --help]"

$ K3S_DEBUG=true  k3s crictl --help 
time="2024-11-13T20:32:06Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
time="2024-11-13T20:32:06Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
time="2024-11-13T20:32:06Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:32:06Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/crictl [crictl --help]"

$ K3S_DEBUG=true  k3s kubectl help 
time="2024-11-13T20:32:12Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:32:12Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl help]"

$ kubectl help 
time="2024-11-13T20:32:17Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:32:17Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl help]"

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml --help 
time="2024-11-13T20:32:22Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:32:22Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml --help]"

$  /usr/local/bin/crictl --help 
time="2024-11-13T20:32:33Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
time="2024-11-13T20:32:33Z" level=warning msg="Failed to stat /home/ubuntu/.rancher/k3s/agent/etc/crictl.yaml: permission denied"
time="2024-11-13T20:32:33Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:32:33Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/crictl [crictl --help]"

$ K3S_DEBUG=true  /usr/local/bin/k3s ctr --help 
time="2024-11-13T20:32:39Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:32:39Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/ctr [ctr --help]"

$ K3S_DEBUG=true sudo k3s ctr --help 
time="2024-11-13T20:32:50Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:32:50Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/ctr [ctr --help]"

$ K3S_DEBUG=true sudo k3s crictl --help 
time="2024-11-13T20:32:56Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:32:56Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/crictl [crictl --help]"

$ K3S_DEBUG=true sudo k3s kubectl help 
time="2024-11-13T20:33:02Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:33:02Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl help]"

$ kubectl help 
time="2024-11-13T20:33:08Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:33:08Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl help]"

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml --help 
time="2024-11-13T20:33:13Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:33:13Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/kubectl [kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml --help]"

$ sudo /usr/local/bin/crictl --help 
time="2024-11-13T20:33:24Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:33:24Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/crictl [crictl --help]"

$ K3S_DEBUG=true sudo /usr/local/bin/k3s ctr --help 
time="2024-11-13T20:33:30Z" level=debug msg="Asset dir /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e"
time="2024-11-13T20:33:30Z" level=debug msg="Running /home/ubuntu/.rancher/k3s/data/251ab4cd583bc4c26de9404b3cf9401f5a79dd4a336bfa8c185563d809d1e87e/bin/ctr [ctr --help]"

$ ls -lrt /home/ubuntu | grep "version" 
$ ls -lrt /home/ubuntu | grep "help"