RKE2 log rotation

[Torkolis]

Hi,

how can I setup log rotation in a RKE2 cluster so that the container logs don't consume all disk space?
https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level

I have found this:
https://www.suse.com/support/kb/doc/?id=000020071
for k3s, can I also apply it to RKE2? Would

curl -sfL https://get.rke2.io | INSTALL_RKE2_EXEC="--kubelet-arg "container-log-max-files=4" --kubelet-arg "container-log-max-size=50Mi"" sh -

work?
Or is there a better way? Can it be achieved for example with some parameters in the file /etc/rancher/rke2/config.yaml?

Edit:
I tried using this in the /etc/rancher/rke2/config.yaml file:

kubelet-arg:
  - container-log-max-files=3
  - container-log-max-size=2Mi

but it seems not to work properly, I can see on the command line (with ps aux) that both parameters are passed to kubelet, but while the files amount is working fine, the log files itself are still large (150 MB) instead of 2, which is also the case without any extra configuration. I don't really understand that, because here https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ it says that default is 10Mi. Do you have any ideas or should I ask in kubelet github repository?

[brandond]

Which specific log files are you looking at?

[Torkolis]

my own simple deployment which produces tons of logs:

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp
data:
  script.sh: | 
    while :; do echo 'Hit CTRL+C'; done
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: ubuntu:22.04
        command: ["bash"]
        args: ["/tmp/script.sh"]
        volumeMounts:
          - name: script
            mountPath: /tmp
      volumes:
        - name: script
          configMap:
            name: myapp

then I watch with
watch ls -al /var/log/pods/<POD>/myapp

[Torkolis]

ok I found it, looks like infinite echo loop without any sleep in it is a bad idea. Just added a sleep 0.02 after the echo in the script.sh and now it works properly.

[DaliborLabudovicGP]

What's the expected behavior? I am assuming that the pod its self does not evict and that the log files are no larger than 50Mb.

[Torkolis]

use this option in the /etc/rancher/rke2/config.yaml file:

kubelet-arg:
  - container-log-max-files=3
  - container-log-max-size=2Mi

defaults for these options are 5 and 10Mi respectively, so one container can produce max 5 log files, 3 of which will be gzipped to be smaller size, so max total size should not exceed 30-40 Mi.

[elaijuh]

Is it possible to configure the rotation of kubelet's log itself? /var/lib/rancher/rke2/agent/logs/kubelet.log

Also for containerd /var/lib/rancher/rke2/agent/containerd/containerd.log

[brandond]

For the kubelet, you can use the legacy kubelet flags to configure logging:

• rke2/pkg/logging/logging.go

  Lines 77 to 83 in 4acdca4

  	logger := &lumberjack.Logger{
  	Filename: filename,
  	MaxSize: int(maxSize),
  	MaxBackups: 3,
  	MaxAge: 28,
  	Compress: true,
  	}

The containerd log rotation settings are hardcoded:

• https://github.com/k3s-io/k3s/blob/8c38d1169d626076256bb0b91bc1322c27b0e63c/pkg/agent/containerd/containerd.go#L47-L53