Better support integration in modern Linux distributions

[AlbanBedel]

Is your feature request related to a problem? Please describe.

k3s is doing several things that goes against standards of modern Linux distributions and make it difficult to integrate:

• Writing to /usr as it might be read only (currently not fatal)
• Writing to /etc as it might be read only (currently fatal)
• Configurations in /var/lib

These issues are particularly problematic with image base distributions where the whole OS might be read only except for /var, /tmp and /run. On such systems the base image generally doesn't include anything under /var which make it difficult to deliver presets, and even more difficult to remove outdated ones. Finally as configurations are only read from /etc or /var/lib it is not possible to distinguish between distribution defaults and configurations from the administrator.

Describe the solution you'd like

k3s would follow modern best practices for its configurations and data storage:

• Configurations are read from /etc, /run, and /usr/lib
• Data such as images are read from /run, /var/lib and /usr/lib
• Persistent data is only written to /var/lib
• Runtime data and configurations is only written to /run

This mean the local administrator config in /etc always has the highest priority, tools can dynamically generate configs in /run and the distro can provide its defaults in /usr/lib. Configs don't really belong in /var/lib but could be included between /run and /usr/lib for backward compatibility.

Data is similar except /var/lib is used for the local data instead of /etc. It should also be expected that entries in /run are probably symlinks to the proper files.

Writing to /etc should be avoided and there should be a fallback in case that is not possible.

Describe alternatives you've considered

Hacks with bind mounts to simulate a writable /etc/rancher, init scripts to copy images from /usr/lib to /var/lib, etc. But that's just painful and not sustainable in the long run.

[brandond]

K3s works on immutable OSs like SLE Micro. Can you name specific distros that behave as you're describing that it does not work on?

[AlbanBedel]

I'm developing a custom distro for embedded devices which is mostly read only, so this feature request stem from my own experience. I have not tested other distros but from a quick search it look like SLE Micro support a writable /etc which I suppose need to be enabled to use k3s without bind mount hacks.

The issue here is not that it is not working at all but rather that it is difficult to maintain long term. For example if my distro want to include an extra manifest it must use a script to copy it from /usr/lib over in /var/lib. Sound simple, but then I want to upgrade it on the next release. What if a user modified this file in between? What if I want to remove it in a later release?

It is not impossible to handle these cases but having a shared directory for the administrator configs, those generated by k3s and those from the distro make it needlessly complex. When each of these actors has its own directory to work with, it make things much easier for everyone. Doubly so if config fragments (aka .d directories) are supported so single config items can be overridden without copying whole configs.

[cwayne18]

Unless we have some evidence that k3s is failing on currently released Modern Linux distros, I don't see anything to do here, we've never claimed to support unreleased custom OS's, and changing the core k3s code to make it work on one could be rather disruptive. Closing this as won't do for now, please re-open if you find issues on released OS's

[brandond]

As a side note, if you're designing a one-off Linux distro from scratch, you might consider ahead of time what applications you want to run on it and design it to work well with those... instead of structuring it in ways that make sense to you, and then going around to application maintainers to try and convince them to change their EXISTING products to work on your new distro.

There's a lot of existing work in this space - flatcar, sle micro, ubuntu core, and so on. To the best of my knowledge, K3s works fine on all of these. You might take a look.

[AlbanBedel]

I submitted this as a feature request to get a feeling if that could be a desirable goal for the project. I hoped get a bit of feedback on the potential issues and challenges before looking at implementing such a thing. A bit sad to see the discussion shutdown down like this.

Lastly I want to point out that what I described in the request is not something I invented, but rather common practice. It even has now been formalized: https://uapi-group.org/specifications/specs/configuration_files_specification/