Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky test: Test_Render_Multiple_Routes #6263

Closed
willdavsmith opened this issue Sep 13, 2023 · 2 comments · Fixed by #6262 or #6333
Closed

Flaky test: Test_Render_Multiple_Routes #6263

willdavsmith opened this issue Sep 13, 2023 · 2 comments · Fixed by #6262 or #6333
Assignees
Labels
bug Something is broken or not working as expected triaged This issue has been reviewed and triaged

Comments

@willdavsmith
Copy link
Contributor

willdavsmith commented Sep 13, 2023

Failed
=== RUN   Test_Render_Multiple_Routes
    render_test.go:1476: 
        	Error Trace:	/home/runner/work/radius/radius/pkg/corerp/renderers/gateway/render_test.go:1476
        	            				/home/runner/work/radius/radius/pkg/corerp/renderers/gateway/render_test.go:875
        	Error:      	Not equal: 
        	            	expected: v1.OutputResource{LocalID:"Gateway", ID:resources.ID{id:"/planes/kubernetes/local/namespaces/test-application/providers/projectcontour.io/HTTPProxy/test-gateway", scopeSegments:[]resources.ScopeSegment{resources.ScopeSegment{Type:"kubernetes", Name:"local"}, resources.ScopeSegment{Type:"namespaces", Name:"test-application"}}, typeSegments:[]resources.TypeSegment{resources.TypeSegment{Type:"projectcontour.io/HTTPProxy", Name:"test-gateway"}}, extensionSegments:[]resources.TypeSegment(nil)}, RadiusManaged:(*bool)(nil), CreateResource:(*v1.Resource)(0xc000180140)}
        	            	actual  : v1.OutputResource{LocalID:"Gateway", ID:resources.ID{id:"/planes/kubernetes/local/namespaces/test-application/providers/projectcontour.io/HTTPProxy/test-gateway", scopeSegments:[]resources.ScopeSegment{resources.ScopeSegment{Type:"kubernetes", Name:"local"}, resources.ScopeSegment{Type:"namespaces", Name:"test-application"}}, typeSegments:[]resources.TypeSegment{resources.TypeSegment{Type:"projectcontour.io/HTTPProxy", Name:"test-gateway"}}, extensionSegments:[]resources.TypeSegment(nil)}, RadiusManaged:(*bool)(nil), CreateResource:(*v1.Resource)(0xc000180000)}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -114,4 +114,4 @@
        	            	   Dependencies: ([]string) (len=2) {
        	            	-   (string) (len=21) "HttpRoute-routenbname",
        	            	-   (string) (len=20) "HttpRoute-routeaname"
        	            	+   (string) (len=20) "HttpRoute-routeaname",
        	            	+   (string) (len=21) "HttpRoute-routenbname"
        	            	   }
        	Test:       	Test_Render_Multiple_Routes
--- FAIL: Test_Render_Multiple_Routes (0.00s)

AB#9337

@willdavsmith willdavsmith added the bug Something is broken or not working as expected label Sep 13, 2023
@kachawla
Copy link
Contributor

This happened again https://github.com/radius-project/radius/actions/runs/6203745054/job/16844827955?pr=6272.

I'm going to label this for Nov release since I think this flakiness is causing ops pain and should be prioritized.

@nithyatsu
Copy link
Contributor

Assigning back to Will since he has a fix by ordering resources in #6262 (comment)

willdavsmith added a commit that referenced this issue Sep 26, 2023
# Description

Today we blindly delete all AWS resources created by our tests at
midnight every day. This could cause functional tests to fail around
this time. This PR adds a `creationTimestamp` to each AWS test resource
that we create as well as logic to delete them by timestamp in the
GitHub workflow.

* Adds AWS creation timestamp to AWS test resources
* Update purge-aws-test-resources GitHub workflow to delete old
resources instead of all resources

## Type of change

<!--

Please select **one** of the following options that describes your
change and delete the others. Clearly identifying the type of change you
are making will help us review your PR faster, and is used in authoring
release notes.

If you are making a bug fix or functionality change to Radius and do not
have an associated issue link please create one now.

-->

- This pull request is a minor refactor, code cleanup, test improvement,
or other maintenance task and doesn't change the functionality of Radius
(issue link optional).

Fixes: #6263

<!--

Please update the following to link the associated issue. This is
required for some kinds of changes (see above).

-->

## Auto-generated summary

<!--
GitHub Copilot for docs will auto-generate a summary of the PR
-->

<!--
copilot:all
-->
### <samp>🤖 Generated by Copilot at adea799</samp>

### Summary
🧹🕒🛠️

<!--
1. 🧹 - This emoji represents the cleanup of old AWS resources and the
improvement of test reliability.
2. 🕒 - This emoji represents the use of creation timestamps to tag and
delete AWS resources based on their age.
3. 🛠️ - This emoji represents the refactoring of the workflow and the
test code to use a common script and a functional package.
-->
This pull request improves the cleanup of AWS resources created by
functional tests. It adds a `creationTimestamp` parameter to the bicep
templates and tags the resources with their creation time. It also
refactors the workflow to use a common script that deletes old resources
based on their tags. It renames a test file and adds some helper
functions to the functional package.

> _We're sailing on the AWS sea, testing our bicep templates_
> _We tag our resources with `creationTimestamp`, so we don't forget_
> _When the tests are done, we run a script to clean up the mess_
> _We heave away, haul away, delete them all on the count of three_

### Walkthrough
* Add a script to delete old AWS resources created by tests
([link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-5625fc3520f54a6e805114f672e280334dac76957f1f5d54cbf3b661f9efeb12R1-R99))
* Update the workflow to use the script instead of inline bash code
([link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-cdbef2239fc7a1aa34d773cb6dae3395193da5015c322ef49bb02025b701a0bcL26-R30),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-cdbef2239fc7a1aa34d773cb6dae3395193da5015c322ef49bb02025b701a0bcL38-R41))
* Add a creationTimestamp parameter to the bicep templates and the
deploy executors for the AWS tests
([link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-531a6f46ae9d530508fc569f3d8efb677f5c2b85ef21732415c71fd370f62050L32-R37),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-531a6f46ae9d530508fc569f3d8efb677f5c2b85ef21732415c71fd370f62050L59-R61),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-531a6f46ae9d530508fc569f3d8efb677f5c2b85ef21732415c71fd370f62050L89-R95),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-531a6f46ae9d530508fc569f3d8efb677f5c2b85ef21732415c71fd370f62050L112-R115),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-b3530585153a4510be532a01249257b09a772f91060a2d1a8be8b3d83fbb8e87L3-R6),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-b3530585153a4510be532a01249257b09a772f91060a2d1a8be8b3d83fbb8e87L9-R20),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-b3530585153a4510be532a01249257b09a772f91060a2d1a8be8b3d83fbb8e87R41-R46),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-e77e6e04f4821e32634940d0dc5c5b2779a7cff9c5c6a0bd27186c64cafbca88L3-R6),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-e77e6e04f4821e32634940d0dc5c5b2779a7cff9c5c6a0bd27186c64cafbca88L9-R20),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-e77e6e04f4821e32634940d0dc5c5b2779a7cff9c5c6a0bd27186c64cafbca88R41-R46),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-ba3f86a7fb8241bd0809f2962fd7561f641229cd4ed710bd00cba541ab66869dR3),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-ba3f86a7fb8241bd0809f2962fd7561f641229cd4ed710bd00cba541ab66869dR15-R18),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-863bee2e7a81fc1dedef482aec1f9d5f965b5a99017411c312d5a6ecf4a34e2cR3),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-863bee2e7a81fc1dedef482aec1f9d5f965b5a99017411c312d5a6ecf4a34e2cR15-R18),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-a6fbd74775c3d83d1c5b066748f75618dea663e00f90a79c8dd115bf452bcbe0L33-R39),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-fbc42064371b122a7ca2d92bfbd5452a51f680c7face11e8db22a71cd7d58e5bL30-R35),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-fbc42064371b122a7ca2d92bfbd5452a51f680c7face11e8db22a71cd7d58e5bL64-R70),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-ea6b32716a499d0deffb7fc13fe99ab1fb5b4fa7f67576e4b89757e76d813b5bL112-R114),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-ea6b32716a499d0deffb7fc13fe99ab1fb5b4fa7f67576e4b89757e76d813b5bL119-R121),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-78e80694bb1ed5b7778407307535be7c50101bfe8b07ee9961e874f4b722c582R3),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-78e80694bb1ed5b7778407307535be7c50101bfe8b07ee9961e874f4b722c582L24-R32),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-43e2af67f3b9b4ceac2bc9597de95219b945ff2a646c5f82dab83a42a3910f6cR3),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-43e2af67f3b9b4ceac2bc9597de95219b945ff2a646c5f82dab83a42a3910f6cR15-R18),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-8028150b2b6a967b9e999213bca3ca236d9428ae62acf3ba5ca2d5d1937fea80R3),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-8028150b2b6a967b9e999213bca3ca236d9428ae62acf3ba5ca2d5d1937fea80R10-R15))
* Add a Tags property to the AWS resources in the bicep templates with
the key 'RadiusCreationTimestamp' and the value of the creationTimestamp
parameter
([link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-b3530585153a4510be532a01249257b09a772f91060a2d1a8be8b3d83fbb8e87L9-R20),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-b3530585153a4510be532a01249257b09a772f91060a2d1a8be8b3d83fbb8e87R41-R46),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-e77e6e04f4821e32634940d0dc5c5b2779a7cff9c5c6a0bd27186c64cafbca88L9-R20),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-e77e6e04f4821e32634940d0dc5c5b2779a7cff9c5c6a0bd27186c64cafbca88R41-R46),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-ba3f86a7fb8241bd0809f2962fd7561f641229cd4ed710bd00cba541ab66869dR15-R18),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-863bee2e7a81fc1dedef482aec1f9d5f965b5a99017411c312d5a6ecf4a34e2cR15-R18),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-78e80694bb1ed5b7778407307535be7c50101bfe8b07ee9961e874f4b722c582L24-R32),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-43e2af67f3b9b4ceac2bc9597de95219b945ff2a646c5f82dab83a42a3910f6cR15-R18),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-8028150b2b6a967b9e999213bca3ca236d9428ae62acf3ba5ca2d5d1937fea80R10-R15))
* Add utility functions to the functional package to get the creation
timestamp and generate a unique S3 bucket name
([link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-d7fffa085c17b99a173ae530015cc075c39419e8ff504aea56ef80eb527b3c18R321-R330))
* Replace the generateS3BucketName function with the one from the
functional package
([link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-fbc42064371b122a7ca2d92bfbd5452a51f680c7face11e8db22a71cd7d58e5bL120-L123))
* Add import statements to use the functional and time packages where
needed
([link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-531a6f46ae9d530508fc569f3d8efb677f5c2b85ef21732415c71fd370f62050R24),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-a6fbd74775c3d83d1c5b066748f75618dea663e00f90a79c8dd115bf452bcbe0L20-R22),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-fbc42064371b122a7ca2d92bfbd5452a51f680c7face11e8db22a71cd7d58e5bL22-R22),
[link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-d7fffa085c17b99a173ae530015cc075c39419e8ff504aea56ef80eb527b3c18R29))
* Rename the file
`test/functional/shared/resources/kubmetadata_cascade_test.go` to
`test/functional/shared/resources/kubemetadata_cascade_test.go`
([link](https://github.com/radius-project/radius/pull/6262/files?diff=unified&w=0#diff-1e27d371cd4d9e9f238ec3ce2d2d567b1fa40156dbe16b83f2f09541c4ecce09))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is broken or not working as expected triaged This issue has been reviewed and triaged
Projects
None yet
4 participants