Incompatible change in patch version breaks downstream/ users of k3s

[meenakshi-koushik]

Environmental Info:
K3s Version:
v1.27.16+k3s1

Node(s) CPU architecture, OS, and Version: Multiple, tested on the following:

Linux node 6.5.0-44-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jun 18 14:36:16 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Linux node 4.18.0-553.8.1.el8_10.x86_64 #1 SMP Tue Jul 2 05:18:08 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server

Describe the bug:
#10500 introduces changes which requires downstream packages e.g., k3d to adapt to new k3s behavior. For example, k3d 5.7.2 works with k3s versions <=1.27.15 and 1.28.1 but not 1.27.16!

Since semantically newer versions oscillate between old and new behavior (1.27.15 -> 1.27.16 -> 1.28.10 -> 1.28.12 -> 1.29.4 -> ...) this is very confusing for users on the upgrade path of k3s.

Steps To Reproduce:
One way to reproduce is to use k3s docker image to setup a cluster using k3d. This fails since the behavior has changed between v1.27.15 and v1.27.16

$ curl -L https://github.com/k3d-io/k3d/releases/download/v5.7.2/k3d-linux-amd64 -o k3d && chmod a+x k3d
$ ./k3d version
k3d version v5.7.2
k3s version v1.29.6-k3s2 (default)
$ ./k3d cluster create test --kubeconfig-update-default=0 --image rancher/k3s:v1.27.15-k3s2 --wait --timeout=240s --no-lb --network test 
$ # works as expected
...
$ ./k3d cluster create test --kubeconfig-update-default=0 --image rancher/k3s:v1.27.16-k3s1 --wait --timeout=240s --no-lb --network test
$ # fails to start cluster
...
$ ./k3d cluster create test --kubeconfig-update-default=0 --image rancher/k3s:v1.28.1-k3s1 --wait --timeout=240s --no-lb --network test 
$ # works as expected, as also  rancher/k3s:v1.28.11-k3s1 and versions in between, but not 1.28.12!

• Installed K3s:

Expected behavior:
Patch versions don't break compatibility or introduce behavioral changes. In other words the following command should be successful (since the same command with 1.27.15 is):

$ ./k3d cluster create test --kubeconfig-update-default=0 --image rancher/k3s:v1.27.16-k3s1 --wait --timeout=240s --no-lb --network test

Actual behavior:
See steps to reproduce above.

Additional context / logs:
It may be possible for k3d to document, for each k3s minor version, the list of patch versions supported by each version of k3d (suggested here). Nevertheless, this change breaks other scripts that use busybox etc. In general these type of changes are expected with minor releases but not with patch versions.

[brandond]

You posted a lot of text here but didn't actually at any point describe the problem, which appears to be that we have dropped the unlinked busybox applet in favor of bundling the findutils version of xargs. Ref: k3s-io/k3s-root#65

This was intended to be a usability improvement, as we were not expecting anyone to be using the unlinked applets directly like this, and we have no particular compatibility contract around use of unlinked applets in our packaged busybox. I am going to close this out, as the issue is in the k3d entrypoint script and has already been resolved.

We do not in general intentionally make breaking changes on patch releases, but sometimes we do bump component versions in order to address security vulnerabilities, and sometimes those bumps will come with slight changes such as this.

this change breaks other scripts that use busybox etc

I'll put it in writing here:

• Don't use unlinked busybox applets from the k3s image.
• We have no compatibility contract around use of busybox applets that aren't linked from the filesystem.

[meenakshi-koushik]

Indeed I only wrote what I encountered and not why it happenned. Thanks for the analysis and pin-pointing the cause. The issue was intended as feedback on the "usability improvement" resulting from the change. It may be useful to capture the actual contract/ api surface in k3s docs or point to it here rather than documenting it here.

[brandond]

I occasionally need to remind myself: