Kubelet flag not supported: shutdownGracePeriod

[jhoelzel]

Environmental Info:
RKE2 Version: latest stable

Node(s) CPU architecture, OS, and Version:
MicroOS latest, amdx64

Cluster Configuration:
Freshly provisioned.

Describe the bug:
Node gracefull shotdown is only possible when providing your own kublet config, which is odd because i can add a number of different configuration values.

Steps To Reproduce:
In the RKE2 Config add

kubelet-arg:
  - "--protect-kernel-defaults=true"
  - "--alsologtostderr=true"
  - "--anonymous-auth=false"
  - "--streaming-connection-idle-timeout=5m"
  - "--eviction-hard=memory.available<250Mi"
  - "--eviction-soft=memory.available<500Mi"
  - "--eviction-soft-grace-period=memory.available=2m"
  - "--kube-reserved=cpu=200m,memory=500Mi"
  - "--system-reserved=cpu=200m,memory=500Mi"
  - "--shutdownGracePeriod=45s"
  - "--shutdownGracePeriodCriticalPods=60s"

Expected behavior:
apply flags without question

Actual behavior:
LOG:
failed to parse kubelet flag: unknown flag: --shutdownGracePeriod

Additional context / logs:
This is of high priority in my opinion, how else would you enable automatic updates with the rancher provided updater?

PS: This can still be achieved by providing your own kubelet conf and applying the path in the config instead, BUT it shouldn't need to

[brandond]

As you noted, graceful shutdown can only be configured by a kubelet configuration file, not via args. You're trying to do it via an arg, which the kubelet correctly reports as unknown. You should create a kubelet configuration file, and reference it from your args:

kubelet-arg:
  - "--protect-kernel-defaults=true"
  - "--alsologtostderr=true"
  - "--anonymous-auth=false"
  - "--streaming-connection-idle-timeout=5m"
  - "--eviction-hard=memory.available<250Mi"
  - "--eviction-soft=memory.available<500Mi"
  - "--eviction-soft-grace-period=memory.available=2m"
  - "--kube-reserved=cpu=200m,memory=500Mi"
  - "--system-reserved=cpu=200m,memory=500Mi"
  - "--config=/etc/rancher/rke2/kubelet.config"

This can still be achieved by providing your own kubelet conf and applying the path in the config instead, BUT it shouldn't need to

It's not up to us to decide what configuration the kubelet supports via CLI flags, and which can only be done via the config file. This is all part of core Kubernetes, not RKE2.

[jhoelzel]

While i see your point, could not all arguments of the kublet-args be written equally the kublet conf?

Right now im basically left wondering if my settings even apply until i provision the cluster.

Another point would be all feature gates that could be used with any kubernetes components:
https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/

[brandond]

could not all arguments of the kublet-args be written equally the kublet conf

Eventually we will put a translation layer in place to allow this, yes.

[jhoelzel]

Want some help?
i belive especially with the CIS profile this might be a gerat idea. Currently you still need to run the benchmarks afterwards in order to find out which settings have been applied as there is no succint list that tells you that. only self assesment currently will reveal all and maybe only for me, thats not declarative enough

[brandond]

Currently you still need to run the benchmarks afterwards in order to find out which settings have been applied as there is no succint list that tells you that.

You can look at the RKE2 logs or ps output to figure out what the settings are, but I agree that's not quite as easy as reading a config file.

@dereknola started in on this a while back, but IIRC we punted on this because:

1. upstream is still adding CLI flags to the kubelet, despite declaring a while back that anything new would only go in the config file and not have a flag.
2. upstream has not shown any indications of actually finishing the job and removing the long-deprecated CLI flags.
3. upstream is not planning on removing CLI flags for any component other than the kubelet.

[jhoelzel]

Should we simply not be able to use the original function footprints for that?

if you check the source https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go
All the config flags are defined directly in AddKubeletConfigFlags

therefore all arguments not present in that setting will be set using arguments instead?