ETCD Backup to S3 - OVHCloud S3-Compatible API

[iwilltry42]

Hey!

I enabled S3 snapshots upload to S3 using the following config, using OVHCloud Object Storage.

  etcd-s3: true
  etcd-s3-endpoint: "s3.sbg.perf.cloud.ovh.net"
  etcd-s3-access-key: "foo"
  etcd-s3-secret-key: "bar"
  etcd-s3-bucket: "mybucket"
  etcd-s3-region: "sbg"
  etcd-s3-folder: "rke2-etcd-backups"

It always just throws "Access Denied" errors, e.g. error retrieving S3 snapshots for reconciliation: Access Denied.

I can use the same credentials without problems though using the AWS CLI or even the MinIO CLI.

Any idea what I could do to debug this?

[brandond]

Can you get any logs from the OVHCloud side? If it is truly S3 compatible, and you've configured the credentials and endpoint correctly, it should work. RKE2 uses the minio golang S3 client libs, which have been confirmed to work with other non-AWS S3-compatible services in the past - we actually use these for QA validation.

[iwilltry42]

I think there's some way to configure access logging there, I'll give that a try.
Good to know that you're using the minio libs, that helps debugging.
Thank you!

[iwilltry42]

OK, that took a while...
I followed this guide to enable access logging on the target bucket: https://help.ovhcloud.com/csm/de-public-cloud-storage-s3-server-access-logging?id=kb_article_view&sysparm_article=KB0056618

I still get the same error and no log entry on OVH side at all.

INFO[0003] Checking if S3 bucket kube-etcd-backups exists 
WARN[0003] Unable to initialize S3 client: Access Denied. 
INFO[0003] Reconciling etcd snapshot data in rke2-etcd-snapshots ConfigMap 
DEBU[0003] core.Factory is initialized                  
DEBU[0003] Attempting to reconcile etcd snapshot data for configmap generation 0 
INFO[0003] Checking if S3 bucket kube-etcd-backups exists 
ERRO[0003] error retrieving S3 snapshots for reconciliation: Access Denied.

[iwilltry42]

OK, I have absolutely no clue why, but I just re-created everything and now it works.
Funny enough, my test program, based on the API example also runs into Access Denied...
But since the original problem is gone, I'll mark this as resolved.