build: Test Models Against the Full Docker Stack

[allgandalf]

Description

In this PR, we have used the full docker stack to test SIPNET by using Docker Compose, this test will run weekly on Mondays.

Motivation and Context

From Discussions in #3184, It would be wise to have a script file and have our cURL POST requests for different models in it as the Matrix option would have to bring up the whole docker stack for each job of the Matrix.

In this PR, we have added the SIPNET model test, which outputs the sipnet.out (model output) file, Test results can be found in this workflow

In the script file, template has been provided to correctly input data into the cURL request so that in future, People can add more models in the script file against which tests can be performed.

Review Time Estimate

• Immediately
• Within one week
• When possible

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My change requires a change to the documentation.
• My name is in the list of CITATION.cff
• I have updated the CHANGELOG.md.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

This PR closes #3184

[allgandalf]

It can be useful to automatically restart the api container if it ever fails when we bring up the whole docker stack

[infotroph]

Why is a prune needed here? Won't it be running in a fresh VM (presumably with ~empty docker cache) every time?

[infotroph]

I'm rusty on the details here -- Does running this test inside the depends container mean these deps are used directly, or do we wind up pulling the whole container again to be run inside itself? Would it work to not specify a container at all let the Docker stack run directly on the host?

[infotroph]

This should probably be a loop over an actual status check; maybe something like while [[ $(docker compose ps --format '{{.State}}'|sort|uniq) != "running" ]]; do sleep 10; done

[infotroph]

How useful is this step for unattended CI run? Will these outputs be helpful for debugging or will they just be filling the log up?

[infotroph]

As above, probably want to loop over a status check and sleep rather than assume a set duration

[infotroph]

These lines seem redundant with the ones above

Suggested change

	# -----------------------------------------------
	# cURL Request Template
	#-----------------------------------------------
	# Be sure to use the -H header and also use the bridge IP address of the Docker host (172.17.0.1) and not localhost
	# curl -v -L -X POST -H "Host: pecan.localhost" \
	# -F 'hostname=' \
	# -F 'modelid=' \
	# -F 'sitegroupid=' \
	# -F 'siteid=' \
	# -F 'sitename=' \
	# -F 'pft[]=' \
	# -F 'start=' \
	# -F 'end=' \
	# -F 'input_met=' \
	# -F 'email=' \
	# -F 'notes=' \
	# 'http://172.17.0.1/pecan/04-runpecan.php'
	# Adjust the sleep time according to the model you are running
	# Append the workflowid and ID after `out/` by 1 for every consecutive run
	# curl -v -L -H "Host: pecan.localhost" \
	# 'http://172.17.0.1/pecan/dataset.php?workflowid=(ID)&type=file&name=out/(ID)/sipnet.out'

[infotroph]

Add a step here verifying that sipnet.out is free from errors / contains the expected output values?

[infotroph]

Left some comments above focused on technical implementation, but from a broader perspective it's worth making sure we're aligned that "Whole-docker-stack Sipnet test via GitHub Actions every Monday" is a good approach. For my part I agree it's smart to make sure we have tests that exercise the whole docker-compose project, but given we already test Sipnet in the daily CI suite I don't have a good sense how much more information we get from testing it again inside the full Docker stack. Emphasis that I really do mean "I don't know", not "I think the proposed approach is wrong"

Pinging @meetagrawal09 for thoughts on how this aligns/interacts with your planned CI development this summer

[meetagrawal09]

@infotroph Some points that come to my mind when I check this implementation are :

1. Should we be more selective as to which containers we are running as part of this full docker stack rather than running all of them at once.

2. We are restricting ourselves with the features provided by the PEcAn API by using curl queries to trigger model runs. A more extensible approach seems to be by using a settings file and triggering workflows by running the workflow.R script.

3. We can run multiple models here as part of the script in the same job but I prefer having separate models to be running as separate jobs. E.g. Here I can run SIPNET and ED2 as part of the shell script but they both are part of the same job. In case SIPNET fails to run correctly this entire job fails without even trying if ED2 was able to run or not. Rather if I configure them as 2 separate jobs we can analyse issues with each model separately.
   By creating a new job for each model run we are separating out concerns with their failures and also reducing time on each run. It also gives us the liberty to choose which model we want to trigger a run for as part of the CI.

[allgandalf]

Closing this PR in favour of #3345, please reopen if you disagree