GPU support

[xhejtman]

RKE2 supports nvidia GPU directly searching for runtime and adding it to containerd config.toml. In our case, however, it does not work because it does not set default runtime value to 'nvidia' (default_runtime_name = "nvidia") as is the case of the gpu-operator from nvidia. Is there any workaround for this? I would not like to change all deployments to request different runtime.

Moreover, the latest nvidia gpu-operator adds runtime nvidia-cdi, so again, I would like to have this one as the default.

It can be set in the config.toml, but looking into sources it does not seem to be templated, so only config.toml.tmpl can be used currently?

[brandond]

If you want that you'd need to provide a custom containerd config template that specifies a different default runtime.

[xhejtman]

Thank you. So nothing planned here so that a default runtime could be specified?

[brandond]

Not at the moment, no. Are you in an environment where a majority of your pods need GPUs? Or you just don't want to have to make any changes to your manifests?

[xhejtman]

The latter. However, we provide 38 node cluster to scientists and I do not want them all to change their deployments.

[brandond]

in the mean time, have you considered a basic mutating admission webhook to set the runtimeClassName on pods that have the GPU env variables set, or using some other matching criteria?

[xhejtman]

I am ok with config template now. It is just something to consider as gpu-operator works this way - setting nvidia as default runtime.