[WIP] Add automated tests

[ma595]

Automatic testing

The tests rely on a job being completed before execution as output data needs to be there for a comparison to happen. For automated testing (regression), this presents a problem. We have therefore shown how it may be possible to use github actions to run the job in advance and then execute a single test.

The automated job needs the following data to execute a test.

• reference data for comparison (zenodo)
• simulation input data.

The zenodo data is approximately 1.1GB. We cannot run SPINacc from scratch because the data dependencies are just too large. This is explained more here

Testing is steered using a .github actions file. .github/workflows/python.yml, and assumes that the initialisation step has already been executed. Therefore auxil.npy and packdata.npy are already there.

It does the following:

• Clones reference data from https://github.com/ma595/SPINacc-data.git including:
  • auxil.npy (metadata) and packdata.npy
  • vlad_data/vlad_data
  • Does some sed to change the execution paths and the test to test_clustering. Equally, we could commit a file and move this to the appropriate location. (Doesn't have to be DEF_Trunk, for example).
  • Change paths in varlist.json to the appropriate path corresponding to the directory of the cloned vlad_data.
  • Runsmain.py
  • pytest then executes tests/test_task1.py only.

Outstanding issues

In order to merge the following issues should be addressed:

• Hardcoded file path in tests directory should be reconsidered.
• Check that the computed and reference outputs correspond
• Encapsulate the above steps in pytest (executing the code shouldn't be separate).
• Fix runtime (reproducibility) tests now that files have been renamed.
• Consider reducing the dataset size and hosting on zenodo. Pull a sample of data to allow pipeline to proceed from end to end (albeit inaccurately).
• rebase

[ma595]

Copying over Tom's comment here:

#45 (comment)

Discussed idea of generating idealized test case for integration test. Not necessarily physically meaningful but something that uses the code in a realistic manner.

@dsgoll123 will make a new small training set for new version ORCHIDEE