Intermittent Backup Failures in Velero with "failed to refresh cached credentials" Error

[halaMayday]

What steps did you take and what happened:

We are using Velero with the velero-plugin-for-aws to perform daily full backups. Every day, we execute backups for over 300 applications, with approximately 10 backups for different applications being executed every 5 minutes. However, we consistently experience occasional backup failures. The error message encountered during these failures is as follows:

time="2024-12-18T06:28:57+08:00" level=error msg="Error getting backup store for this location" backupLocation=xxxxxx controller=backup-sync error="rpc error: code = Unknown desc = failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded" error.file="velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:150" error.function="main.(*ObjectStore).Init" logSource="pkg/controller/backup_sync_controller.go:111"

This issue consistently happens every day, affecting a small number of backups.

Environment:

• Velero version: 1.13.1
• Velero plugin for AWS version: 1.9.1

[kaovilai]

Set the AWS env vars and then trick it into using an empty profile, eg:

terraformer import aws -r s3 --profile ""

Found this note here. Can you try setting BSL profile to empty?