Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AWS creation timestamp to test resources #6262

Merged
merged 11 commits into from
Sep 26, 2023

Conversation

willdavsmith
Copy link
Contributor

@willdavsmith willdavsmith commented Sep 13, 2023

Description

Today we blindly delete all AWS resources created by our tests at midnight every day. This could cause functional tests to fail around this time. This PR adds a creationTimestamp to each AWS test resource that we create as well as logic to delete them by timestamp in the GitHub workflow.

  • Adds AWS creation timestamp to AWS test resources
  • Update purge-aws-test-resources GitHub workflow to delete old resources instead of all resources

Type of change

  • This pull request is a minor refactor, code cleanup, test improvement, or other maintenance task and doesn't change the functionality of Radius (issue link optional).

Fixes: #6263

Auto-generated summary

🤖 Generated by Copilot at adea799

Summary

🧹🕒🛠️

This pull request improves the cleanup of AWS resources created by functional tests. It adds a creationTimestamp parameter to the bicep templates and tags the resources with their creation time. It also refactors the workflow to use a common script that deletes old resources based on their tags. It renames a test file and adds some helper functions to the functional package.

We're sailing on the AWS sea, testing our bicep templates
We tag our resources with creationTimestamp, so we don't forget
When the tests are done, we run a script to clean up the mess
We heave away, haul away, delete them all on the count of three

Walkthrough

  • Add a script to delete old AWS resources created by tests (link)
  • Update the workflow to use the script instead of inline bash code (link, link)
  • Add a creationTimestamp parameter to the bicep templates and the deploy executors for the AWS tests (link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link)
  • Add a Tags property to the AWS resources in the bicep templates with the key 'RadiusCreationTimestamp' and the value of the creationTimestamp parameter (link, link, link, link, link, link, link, link, link)
  • Add utility functions to the functional package to get the creation timestamp and generate a unique S3 bucket name (link)
  • Replace the generateS3BucketName function with the one from the functional package (link)
  • Add import statements to use the functional and time packages where needed (link, link, link, link)
  • Rename the file test/functional/shared/resources/kubmetadata_cascade_test.go to test/functional/shared/resources/kubemetadata_cascade_test.go (link)

@willdavsmith willdavsmith requested a review from a team as a code owner September 13, 2023 16:55
@github-actions
Copy link

github-actions bot commented Sep 13, 2023

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref refs/pull/6262/merge
Unique ID cfe3f91e72
Image tag pr-cfe3f91e72
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.11.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location radiusdev.azurecr.io/test/functional/shared/recipes/<name>:pr-cfe3f91e72
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-cfe3f91e72
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-cfe3f91e72
  • deployment-engine test image location: radius.azurecr.io/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting shared functional tests...
⌛ Starting daprrp functional tests...
⌛ Starting ucp functional tests...
⌛ Starting msgrp functional tests...
⌛ Starting samples functional tests...
⌛ Starting datastoresrp functional tests...
✅ datastoresrp functional tests succeeded
✅ msgrp functional tests succeeded
✅ ucp functional tests succeeded
✅ samples functional tests succeeded
✅ daprrp functional tests succeeded
❌ shared functional test failed. Please check the logs for more details

@github-actions
Copy link

github-actions bot commented Sep 13, 2023

Test Results

2 976 tests  ±0   2 967 ✔️ ±0   2m 37s ⏱️ +23s
   252 suites ±0          9 💤 ±0 
       1 files   ±0          0 ±0 

Results for commit 3b7d40d. ± Comparison against base commit cf80402.

This pull request removes 2 and adds 2 tests. Note that renamed tests count towards both.
github.com/radius-project/radius/pkg/armrpc/frontend/controller ‑ TestValidateEtag_IfMatch/c3ba4e2c-c7e7-4d39-8956-bb0636200722
github.com/radius-project/radius/pkg/armrpc/frontend/controller ‑ TestValidateEtag_IfMatch/c3ba4e2c-c7e7-4d39-8956-bb0636200722#01
github.com/radius-project/radius/pkg/armrpc/frontend/controller ‑ TestValidateEtag_IfMatch/45644278-41ae-4e5f-b5a8-77609f9289b7
github.com/radius-project/radius/pkg/armrpc/frontend/controller ‑ TestValidateEtag_IfMatch/45644278-41ae-4e5f-b5a8-77609f9289b7#01

♻️ This comment has been updated with latest results.

@github-actions
Copy link

66.3

For the detailed report, please go to Checks tab, click Build and Test, and then download unit_test_coverage artifact at the bottom of build page.

  • Your PR branch coverage: 66.3 %
  • main branch coverage: 66.3 %
  • diff coverage: 0 %

The coverage result does not include the functional test coverage.

@github-actions
Copy link

github-actions bot commented Sep 13, 2023

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref refs/pull/6262/merge
Unique ID cb24586b78
Image tag pr-cb24586b78
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.11.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location radiusdev.azurecr.io/test/functional/shared/recipes/<name>:pr-cb24586b78
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-cb24586b78
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-cb24586b78
  • deployment-engine test image location: radius.azurecr.io/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting datastoresrp functional tests...
⌛ Starting msgrp functional tests...
⌛ Starting samples functional tests...
⌛ Starting ucp functional tests...
⌛ Starting shared functional tests...
⌛ Starting daprrp functional tests...
✅ datastoresrp functional tests succeeded
✅ samples functional tests succeeded
✅ ucp functional tests succeeded
✅ msgrp functional tests succeeded
✅ daprrp functional tests succeeded
❌ shared functional test failed. Please check the logs for more details

@github-actions
Copy link

66.2

For the detailed report, please go to Checks tab, click Build and Test, and then download unit_test_coverage artifact at the bottom of build page.

  • Your PR branch coverage: 66.2 %
  • main branch coverage: 66.3 %
  • diff coverage: -.1 %

The coverage result does not include the functional test coverage.

@github-actions
Copy link

github-actions bot commented Sep 13, 2023

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref refs/pull/6262/merge
Unique ID 8c5b6b8c7b
Image tag pr-8c5b6b8c7b
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.11.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location radiusdev.azurecr.io/test/functional/shared/recipes/<name>:pr-8c5b6b8c7b
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-8c5b6b8c7b
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-8c5b6b8c7b
  • deployment-engine test image location: radius.azurecr.io/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting ucp functional tests...
⌛ Starting datastoresrp functional tests...
⌛ Starting msgrp functional tests...
⌛ Starting shared functional tests...
⌛ Starting samples functional tests...
⌛ Starting daprrp functional tests...
✅ datastoresrp functional tests succeeded
✅ ucp functional tests succeeded
✅ msgrp functional tests succeeded
✅ samples functional tests succeeded
✅ daprrp functional tests succeeded
❌ shared functional test failed. Please check the logs for more details

@github-actions
Copy link

github-actions bot commented Sep 13, 2023

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref refs/pull/6262/merge
Unique ID 40d32b4d01
Image tag pr-40d32b4d01
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.11.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location radiusdev.azurecr.io/test/functional/shared/recipes/<name>:pr-40d32b4d01
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-40d32b4d01
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-40d32b4d01
  • deployment-engine test image location: radius.azurecr.io/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting samples functional tests...
⌛ Starting msgrp functional tests...
⌛ Starting datastoresrp functional tests...
⌛ Starting shared functional tests...
⌛ Starting ucp functional tests...
✅ datastoresrp functional tests succeeded
✅ msgrp functional tests succeeded
✅ samples functional tests succeeded
✅ ucp functional tests succeeded
✅ daprrp functional tests succeeded
❌ shared functional test failed. Please check the logs for more details

@github-actions
Copy link

github-actions bot commented Sep 13, 2023

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref refs/pull/6262/merge
Unique ID d9b73ee4b6
Image tag pr-d9b73ee4b6
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.11.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location radiusdev.azurecr.io/test/functional/shared/recipes/<name>:pr-d9b73ee4b6
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-d9b73ee4b6
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-d9b73ee4b6
  • deployment-engine test image location: radius.azurecr.io/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting datastoresrp functional tests...
⌛ Starting ucp functional tests...
⌛ Starting msgrp functional tests...
⌛ Starting samples functional tests...
✅ datastoresrp functional tests succeeded
✅ msgrp functional tests succeeded
✅ ucp functional tests succeeded
✅ samples functional tests succeeded
✅ daprrp functional tests succeeded
❌ shared functional test failed. Please check the logs for more details

@github-actions
Copy link

66.3

For the detailed report, please go to Checks tab, click Build and Test, and then download unit_test_coverage artifact at the bottom of build page.

  • Your PR branch coverage: 66.3 %
  • main branch coverage: 66.3 %
  • diff coverage: 0 %

The coverage result does not include the functional test coverage.

@github-actions
Copy link

github-actions bot commented Sep 14, 2023

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref refs/pull/6262/merge
Unique ID ed7883335c
Image tag pr-ed7883335c
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.11.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location radiusdev.azurecr.io/test/functional/shared/recipes/<name>:pr-ed7883335c
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-ed7883335c
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-ed7883335c
  • deployment-engine test image location: radius.azurecr.io/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting datastoresrp functional tests...
⌛ Starting ucp functional tests...
⌛ Starting shared functional tests...
⌛ Starting msgrp functional tests...
⌛ Starting daprrp functional tests...
✅ datastoresrp functional tests succeeded
✅ ucp functional tests succeeded
✅ samples functional tests succeeded
✅ msgrp functional tests succeeded
✅ daprrp functional tests succeeded
✅ shared functional tests succeeded

@github-actions
Copy link

66.3

For the detailed report, please go to Checks tab, click Build and Test, and then download unit_test_coverage artifact at the bottom of build page.

  • Your PR branch coverage: 66.3 %
  • main branch coverage: 66.3 %
  • diff coverage: 0 %

The coverage result does not include the functional test coverage.

@@ -1473,6 +1474,10 @@ func validateHTTPProxy(t *testing.T, outputResources []rpv1.OutputResource, expe
}
}

// Sort the dependencies so that tests aren't flaky
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

separate issue, not related to the rest of the PR: #6263

@github-actions
Copy link

github-actions bot commented Sep 21, 2023

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref refs/pull/6262/merge
Unique ID 7376554268
Image tag pr-7376554268
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.11.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location radiusdev.azurecr.io/test/functional/shared/recipes/<name>:pr-7376554268
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-7376554268
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-7376554268
  • deployment-engine test image location: radius.azurecr.io/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting datastoresrp functional tests...
⌛ Starting ucp functional tests...
⌛ Starting shared functional tests...
⌛ Starting samples functional tests...
⌛ Starting daprrp functional tests...
✅ datastoresrp functional tests succeeded
✅ ucp functional tests succeeded
✅ msgrp functional tests succeeded
✅ samples functional tests succeeded
✅ daprrp functional tests succeeded
✅ shared functional tests succeeded

@github-actions
Copy link

66.2

For the detailed report, please go to Checks tab, click Build and Test, and then download unit_test_coverage artifact at the bottom of build page.

  • Your PR branch coverage: 66.2 %
  • main branch coverage: 66.2 %
  • diff coverage: 0 %

The coverage result does not include the functional test coverage.

@willdavsmith willdavsmith requested review from a team as code owners September 22, 2023 14:20
@github-actions
Copy link

github-actions bot commented Sep 22, 2023

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref refs/pull/6262/merge
Unique ID 446906d576
Image tag pr-446906d576
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.11.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location radiusdev.azurecr.io/test/functional/shared/recipes/<name>:pr-446906d576
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-446906d576
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-446906d576
  • deployment-engine test image location: radius.azurecr.io/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting datastoresrp functional tests...
⌛ Starting msgrp functional tests...
⌛ Starting samples functional tests...
⌛ Starting daprrp functional tests...
⌛ Starting ucp functional tests...
⌛ Starting shared functional tests...
✅ datastoresrp functional tests succeeded
✅ msgrp functional tests succeeded
✅ samples functional tests succeeded
✅ ucp functional tests succeeded
✅ daprrp functional tests succeeded
✅ shared functional tests succeeded

@github-actions
Copy link

66.2

For the detailed report, please go to Checks tab, click Build and Test, and then download unit_test_coverage artifact at the bottom of build page.

  • Your PR branch coverage: 66.2 %
  • main branch coverage: 66.2 %
  • diff coverage: 0 %

The coverage result does not include the functional test coverage.

@github-actions
Copy link

github-actions bot commented Sep 25, 2023

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref refs/pull/6262/merge
Unique ID 9eff7f85e0
Image tag pr-9eff7f85e0
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.11.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location radiusdev.azurecr.io/test/functional/shared/recipes/<name>:pr-9eff7f85e0
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-9eff7f85e0
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-9eff7f85e0
  • deployment-engine test image location: radius.azurecr.io/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting datastoresrp functional tests...
⌛ Starting ucp functional tests...
⌛ Starting shared functional tests...
⌛ Starting msgrp functional tests...
⌛ Starting daprrp functional tests...
⌛ Starting samples functional tests...
✅ datastoresrp functional tests succeeded
✅ ucp functional tests succeeded
✅ msgrp functional tests succeeded
✅ samples functional tests succeeded
✅ daprrp functional tests succeeded
✅ shared functional tests succeeded

@github-actions
Copy link

66.2

For the detailed report, please go to Checks tab, click Build and Test, and then download unit_test_coverage artifact at the bottom of build page.

  • Your PR branch coverage: 66.2 %
  • main branch coverage: 66.2 %
  • diff coverage: 0 %

The coverage result does not include the functional test coverage.

@github-actions
Copy link

github-actions bot commented Sep 26, 2023

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref refs/pull/6262/merge
Unique ID 9fdf95e52a
Image tag pr-9fdf95e52a
Click here to see the list of tools in the current test run
  • gotestsum 1.10.0
  • KinD: v0.20.0
  • Dapr: 1.11.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.1.0
  • Bicep recipe location radiusdev.azurecr.io/test/functional/shared/recipes/<name>:pr-9fdf95e52a
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-9fdf95e52a
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-9fdf95e52a
  • deployment-engine test image location: radius.azurecr.io/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting datastoresrp functional tests...
⌛ Starting ucp functional tests...
⌛ Starting msgrp functional tests...
⌛ Starting samples functional tests...
⌛ Starting daprrp functional tests...
✅ datastoresrp functional tests succeeded
✅ ucp functional tests succeeded
✅ samples functional tests succeeded
✅ msgrp functional tests succeeded
✅ daprrp functional tests succeeded
✅ shared functional tests succeeded

@github-actions
Copy link

66.2

For the detailed report, please go to Checks tab, click Build and Test, and then download unit_test_coverage artifact at the bottom of build page.

  • Your PR branch coverage: 66.2 %
  • main branch coverage: 66.2 %
  • diff coverage: 0 %

The coverage result does not include the functional test coverage.

@willdavsmith willdavsmith merged commit 240ae75 into main Sep 26, 2023
@willdavsmith willdavsmith deleted the willdavsmith/aws-creation-timestamp branch September 26, 2023 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flaky test: Test_Render_Multiple_Routes
2 participants