How to customize etcd backup configuration in config.yaml (backup to s3)

[0asp0]

Hi, I've setup an rke2 cluster with 3 nodes. For installation I used the script available at https://get.rke2.io.

Now I want to customize the etcd backup. Beside saving the backup on local disk I want the backup to be shipped to an internal minio s3 service.

I was able to get an onDemand Backup shipped to s3 using the following command on console:

rke2 etcd-snapshot \
--etcd-s3 \
--etcd-s3-bucket=k8s-local-backup \
--etcd-s3-access-key=myuser \
--etcd-s3-secret-key=mypassword \
--etcd-s3-folder=myfolder \
--etcd-s3-endpoint=minio.local:9000 \
--etcd-s3-endpoint-ca=/etc/pki/ca-trust/source/anchors/my-ca.pem

Now I want to add this configuration to the config yaml, but all tries failed. My settings have been ignored, so I assume I made an mistake, using wrong keys or whatever.
What did I do?

• I only processed one node of the cluster
• drain the node
• change the config
• restart the node
• uncordon the node

Here is my config snipped which I added to the config which does NOT work:

etcd-s3: true
etcd-snapshot-schedule-cron value: '*/5 * * * *'
etcd-s3-bucket: k8s-local-backup
etcd-s3-access-key: myuser
etcd-s3-secret-key: mykey
etcd-s3-folder: myfolder
etcd-s3-endpoint: minio.local:9000
etcd-s3-endpoint-ca: /etc/pki/ca-trust/source/anchors/my-ca.pem

also this variant does NOT work:

etcd:
  disableSnapshots: false
  s3:
    bucket: k8s-local-backup
    endpoint: minio.local:9000
    endpointCA: /etc/pki/ca-trust/source/anchors/my-ca.pem
    folder: myfolder
    accessKey: myuser
    secretKey: mykey
  snapshotRetention: 5
  snapshotScheduleCron: '0 * * * *'

Is there a way to config-check the config.yaml on known field basis, not just that is a correct yaml? (rke2-server will not start if it is invalid yaml format)

Can you please point out the correct configuration snippet which I need to insert into config.yaml?

Thanks, Andreas

[brandond]

Is there a way to config-check the config.yaml on known field basis, not just that is a correct yaml?

No. The config file does not have a strict schema. Internally the key-value pairs are just converted to CLI flags and values and passed into the normal flag parser.

Here is my config snipped which I added to the config which does NOT work:

Does not work how? Do you get errors, or does it just not seem to be taking effect? Did you restart the rke2-server service after updating the config file? The etcd-snapshot subcommand will read the config file when you run it manually, but for scheduled snapshots to use that config, you need to restart the service.

[0asp0]

yes, I restarted the service after putting in the configuration.

I made it a step further. After checking today I saw automatic generated backups which have been pushed to s3.

So my main issue was that I made a mistake while changing the cron pattern:

etcd-snapshot-schedule-cron value: '*/5 * * * *'

must be

etcd-snapshot-schedule-cron: '*/5 * * * *'

resulting in the complete config template for exporting etcd each 5 minutes (for testing purpose):

etcd-s3: true
etcd-snapshot-schedule-cron: '*/5 * * * *'
etcd-s3-bucket: k8s-local-backup
etcd-s3-access-key: myuser
etcd-s3-secret-key: mykey
etcd-s3-folder: myfolder
etcd-s3-endpoint: minio.local:9000
etcd-s3-endpoint-ca: /etc/pki/ca-trust/source/anchors/my-ca.pem

btw. If anyone is wondering where to see some log output for the etcd backup:
I saw some log entries with journalctl -u rke2-server on the host.

Also a summary of the backup is stored in a configmap inside the cluster. You can access it by kubectl -n kube-system get configmap rke2-etcd-snapshots -oyaml

Regards, Andreas