Development: Make custom cloud testing simple to deploy and maintain

[tmbgreaves]

Some research software will require more testing capability than is available from a fully hosted and managed platform such as Azure Pipelines. In these cases, it's useful to be able to deploy a custom testing platform into the cloud. Whilst there is already a lot of detailed documentation about deploying custom CI to the cloud, it often makes assumptions that are very valid in the case of applications managed by people with a technical systems background but which are not well suited to research software projects.

Key considerations from a research software perspective are:

• All required resources provided by cloud services
• Simple deployment -- no systems management skills required
• No ongoing systems management required for maintenance (host upgrades set to automatic, testing platform upgrades handled automatically or from inside the testing interface)
• No state saved on the master host (it should be able to be discarded and recreated trivially)
• All testing workers deployed on-demand
• Testing platform internal state saved to cloud storage
• Deployment/redeployment templated and possible from command line