[TEST] TestBackupRestore::test_connection[S3] test fails making consecutive tests fail for backup & restore

[khushboo-rancher]

What's the test to develop? Please describe

 @pytest.mark.dependency()
    def test_connection(self, api_client, backup_config, config_backup_target):
        code, data = api_client.settings.backup_target_test_connection()
>       assert 200 == code, f'Failed to test backup target connection: {data}'
E       AssertionError: Failed to test backup target connection: {"errors":["can't connect to backup target \u0026{Type:s3 Endpoint:http://172.19.99.205:9000 AccessKeyID:minioadmin SecretAccessKey:minioadmin BucketName:giants BucketRegion:ecmlab Cert: VirtualHostedStyle:false}, error: secrets \"harvester-backup-target-secret\" not found"]}
E       assert 200 == 503

api_client = HarvesterAPI('https://172.19.xyz.x ...

[irishgordo]

at first brief audit everything looks good on the connection side.
the earlier building of the secret is failing probably... still investigating
secrets \"harvester-backup-target-secret\" not found"

[noahgildersleeve]

I retested this with a local environment with v1.4.0-rc1 and it all passed. However I had some issues when trying to run the test with other tests, that might be unrelated though.

[irishgordo]

@noahgildersleeve - 🤔 - hmmm, I can consistently reproduce this issue with v1.4.0-rc1 locally -> w/ a separate Harvester bare-metal cluster.
It seems to be a timing issue.

I've opened up a PR to fix this:
#1580

Demonstrated in video below with:

• 2 Node Bare-Metal External V1.4.0-rc1 Harvester Cluster HP DL160s
• Just running tests locally w/ Python3.10 venv setup in VSCode
• A minio instance running with TLS/SSL via Docker locally, additional-ca added to Harvester side

s3-connection-building-issue.mp4