Improve functional tests

[sk593]

Overview of feature request

Today we test end-to-end deployments for Radius resources using functional tests. While this is a great way to verify the functionality of resources, there's definitely room for improvement.

First, the structure of the tests is extremely hard to navigate. We have no separation between the different resources and namespaces. Applications.Core and Applications.Link resources all reside in the same folder. To make the structure easier to follow, we should separate resources by namespace. Now that we're adding AWS functional tests and will eventually have support for Terraform, one step further would be to separate tests by provider as well.

The contribution guides also need to be updated. These haven't been updated for almost a year and as we move to open source, it'll be important to have good documentation on how to write these tests, how to run them, how they're run on Github, etc.

We should also investigate ways to improve the time it takes to run the tests. There were recently changes to the repo that sped up the tests but we're still experiencing long wait times while waiting for the functional tests to complete. This has been a huge pain point for development work since there's a lot of down time. Branches are also required to be updated with main before a merge can happen and the tests can run which causes a lot of delays when multiple developers are trying to merge in changes.

Please feel free add any other suggestions or comments to this issue

Acceptance criteria

• Separate resources by namespace (and potentially by provider)
• Update contribution guides
• Investigate speeding up functional test runs

Additional context

AB#8388

AB#9506

[rynowak]

Meeting notes 6/20

We identified the following topics to dig into, and then focused on the test matrix as the main topic since it feeds into the others.

• Structure and organization
• Isolation and framework
• What is our test matrix?
  • Guidance on whether to use pre-created cloud resources? Which tests use cloud resource?
  • Negative testing - clear differentiation between functional tests and feature level tests
  • Which tests should run as part of a PR?
• Test speed and duration
  • Requirement to rerun the tests when another change is merged
  • Test run takes ~30-40 minutes

The Matrix

• Some basic installation tests
• E2E application deployment scenarios that users perform
  • eg: Creating an application
  • eg: Create an application that uses a recipe (eg: redis)
    • Deploy a container that connects to Redis
    • Use a "test recipe" that deploys Redis (eg: run Redis in a container on Kubernetes)
    • Current guidance: copy from existing test and change it
    • Current guidance: one for manual mode, one for Recipe mode using self-hosted Kubernetes resources
    • Current guidance: organized by resource (2 scenarios per resource)
  • Deletion is included
  • Feature-level tests
    • IAM role configuration
    • Kubernetes metadata
    • List Secrets
    • Recipe selection logic
    • Recipe parameters
    • Recipe integration with cloud resources
      • We have tests for mongo with Azure resources
      • We are building a test for that uses AWS resource (S3)
  • Terraform vs Bicep (what kind of recipe)
• Negative testing and error handling
  • eg: how to do we handle errors that come back from cloud providers
  • focus on error messages and reliability is especially valuable to users
• Tests for recipes themselves in project-radius/recipes
  • We will define a contribution process for these recipes
  • We will need automation as part of that
  • Like we have automation for samples, we also want to have tests for the recipes
• Performance tests
• Long haul testing and metrics
• Security tests

We can't/don't test every resource with cloud resources. It would take too long, but we also run the risk of missing things like timeouts or failures.

Next Steps

We want to create a proposal that answers the follow questions as a next step:

• What is our test matrix?
  • Guidance on whether to use pre-created cloud resources? Which tests use cloud resource?
  • Negative testing - clear differentiation between functional tests and feature level tests
  • Which tests should run as part of a PR?
  • Define structure based on decisions about test matrix?
  • What do we want to test and how should we test it?
    • Based on that, how we can optimize the contributor experience?
    • How do we prioritize tests? What are the most important tests to have in place?