Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perform unit tests in NSLS-II conda environments #96

Merged
merged 46 commits into from
Jun 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
4c3d993
Add conda-env-tests workflow
padraic-shafer May 21, 2024
d0dd220
Activate incubator conda environment
padraic-shafer May 21, 2024
b08833e
Include archived conda env checksums
padraic-shafer May 21, 2024
90410ab
Cache the conda environment archive
padraic-shafer May 21, 2024
a0a24aa
Record the conda environment used for installation and test
padraic-shafer May 21, 2024
d1c0cb2
Fix cache path for archived conda environment file
padraic-shafer May 21, 2024
771d2b8
Activate the NSLS-II conda environment for installation and test
padraic-shafer May 21, 2024
c7d3e31
Cache the archive conda env
padraic-shafer May 21, 2024
9eb6a00
Do not upgrade numpy for conda env tests
padraic-shafer May 21, 2024
800c801
Move conda env details to GHA matrix strategy
padraic-shafer May 21, 2024
e2fad9e
Fix quotes in GHA env vars
padraic-shafer May 21, 2024
240573e
Tests with non-tiled conda envs
padraic-shafer May 21, 2024
b0b9940
Add all 2023 and 2024 conda envs
padraic-shafer May 21, 2024
69df7be
Limit number of concurrent GHA jobs
padraic-shafer May 21, 2024
2c677b5
Fix zenodo id for 2023-1.1
padraic-shafer May 21, 2024
ba97b04
Workaround for unset GDAL_DATA variable
padraic-shafer May 22, 2024
97f0e4c
Workaround #2 for unset GDAL_DATA variable
padraic-shafer May 22, 2024
db625ab
Workaround for unset GDAL_DRIVER_PATH variable
padraic-shafer May 22, 2024
f5101ef
Skip passing tests for now
padraic-shafer May 22, 2024
ac1aedd
Workaround for unset GEOTIFF_CSV variable
padraic-shafer May 22, 2024
9f53424
Workaround for unset MKL_INTERFACE_LAYER variable
padraic-shafer May 22, 2024
47cb4d1
Tidy up conda-envs-test config
padraic-shafer May 22, 2024
73c10db
Add missing conda env 2023-1.2-py310-tiled
padraic-shafer May 22, 2024
1cc9f9b
Build conda-env-tests matrix from YAML input
padraic-shafer May 22, 2024
37993cf
Define job dependency in conda-env-tests
padraic-shafer May 22, 2024
0b449cd
Use the job dependency in conda-env-tests
padraic-shafer May 22, 2024
bd4856c
Use python script to import test matrix
padraic-shafer May 22, 2024
7a86ea1
Reference output ot matrix_prep job
padraic-shafer May 22, 2024
fb4bf8f
Parse JSON input for test strategy matrix
padraic-shafer May 22, 2024
48ac597
Refactor test-in-conda-env as reusable workflow
padraic-shafer May 22, 2024
6ad70f7
Use branch name to version the workflow action
padraic-shafer May 22, 2024
c57777a
Fix env usage
padraic-shafer May 22, 2024
8d68440
Remove spaces from variable expansion
padraic-shafer May 23, 2024
bee124e
Move reusable workflows into workflows folder
padraic-shafer May 23, 2024
7293beb
Try actions rather than workflows
padraic-shafer May 23, 2024
53d2156
Debug env var
padraic-shafer May 23, 2024
0499b41
Debug env var isolated
padraic-shafer May 23, 2024
1299368
Revert to workflows for dispatched workflow
padraic-shafer May 23, 2024
7b9bd87
Checkout code for local action
padraic-shafer May 23, 2024
7618e48
Revert to single workflow step
padraic-shafer May 23, 2024
e941a6a
Pass JSON string, not sequence to workflow
padraic-shafer May 23, 2024
b2688ca
Extract matrix strategy from inner to outer workflow
padraic-shafer May 23, 2024
b7c38ca
Extract matrix_prep to a reusable workflow
padraic-shafer May 23, 2024
6ccf751
Do not define type of GHA workflow output
padraic-shafer May 23, 2024
e929218
Split long-running legacy tests into separate GHA
padraic-shafer May 23, 2024
c2aaa07
Rename long-running legacy tests
padraic-shafer May 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 140 additions & 0 deletions .github/conda-env-tests/legacy-envs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
---

# 2023-1.0
- conda_env_name: '2023-1.0-py39'
zenodo_id: '7470912'
md5_checksum: '651be0e26cdb230dd8c809b4eed046e9'
python-version: '3.9'

- conda_env_name: '2023-1.0-py39-tiled'
zenodo_id: '7470912'
md5_checksum: 'c092b113d19510ac9ecb68eda66d3e79'
python-version: '3.9'

- conda_env_name: '2023-1.0-py310'
zenodo_id: '7470912'
md5_checksum: '2c93e892a55d8c684acea4ccb2680ac5'
python-version: '3.10'

- conda_env_name: '2023-1.0-py310-tiled'
zenodo_id: '7470912'
md5_checksum: '1de3a83f0ead58d3cb60d721478748a2'
python-version: '3.10'

# 2023-1.1
- conda_env_name: '2023-1.1-py39'
zenodo_id: '7603440'
md5_checksum: '531a712033d907b781d24030182357f4'
python-version: '3.9'

- conda_env_name: '2023-1.1-py39-tiled'
zenodo_id: '7603440'
md5_checksum: '31dad55d53436605c47dbfc510e876a2'
python-version: '3.9'

- conda_env_name: '2023-1.1-py310'
zenodo_id: '7603440'
md5_checksum: 'd60ceb752a798590334cbec3d0155fb6'
python-version: '3.10'

- conda_env_name: '2023-1.1-py310-tiled'
zenodo_id: '7603440'
md5_checksum: 'f501314939d5237edb58244b0fcc6857'
python-version: '3.10'

# 2023-1.2
- conda_env_name: '2023-1.2-py39'
zenodo_id: '7651500'
md5_checksum: '14d0629f468df311918cfea2e6413bfa'
python-version: '3.9'

- conda_env_name: '2023-1.2-py39-tiled'
zenodo_id: '7651500'
md5_checksum: '31586643a0cdd9aee8349ccb93ccac87'
python-version: '3.9'

- conda_env_name: '2023-1.2-py310'
zenodo_id: '7651500'
md5_checksum: '2dc85c4d2392f751c76ed7d01f363814'
python-version: '3.10'

- conda_env_name: '2023-1.2-py310-tiled'
zenodo_id: '7651500'
md5_checksum: 'ff8c0c475a007434c0011f5d8976d853'
python-version: '3.10'

# 2023-1.3
- conda_env_name: '2023-1.3-py39'
zenodo_id: '7791542'
md5_checksum: '9cc18f3e09dfa30c4e8f165a932ca394'
python-version: '3.9'

- conda_env_name: '2023-1.3-py39-tiled'
zenodo_id: '7791542'
md5_checksum: '1f40f5786fc0c06baad3fd3b456a6abc'
python-version: '3.9'

- conda_env_name: '2023-1.3-py310'
zenodo_id: '7791542'
md5_checksum: '3304bc674d14b2820d60b311b58661fd'
python-version: '3.10'

- conda_env_name: '2023-1.3-py310-tiled'
zenodo_id: '7791542'
md5_checksum: 'b7869b9fdb1c076bfba1f7eb2c63c6f0'
python-version: '3.10'

# 2023-2.0
- conda_env_name: '2023-2.0-py310'
zenodo_id: '7876026'
md5_checksum: '2f81225554a16339e2a49432cd8a013b'
python-version: '3.10'

- conda_env_name: '2023-2.0-py310-tiled'
zenodo_id: '7876026'
md5_checksum: 'ef85af9ff76bd0afa527bdfc391b0442'
python-version: '3.10'

# 2023-2.1
- conda_env_name: '2023-2.1-py310'
zenodo_id: '8098505'
md5_checksum: '31add8231d47d92821b5f4178cd9a58a'
python-version: '3.10'

- conda_env_name: '2023-2.1-py310-tiled'
zenodo_id: '8098505'
md5_checksum: 'd75299b2a6ffc2ed5aca7d01150c97f0'
python-version: '3.10'

# 2023-3.0
- conda_env_name: '2023-3.0-py310'
zenodo_id: '8342731'
md5_checksum: '49ad184de893ab1a27db3c4d3e91a045'
python-version: '3.10'

- conda_env_name: '2023-3.0-py310-tiled'
zenodo_id: '8342731'
md5_checksum: 'b727a21d791192bc3981fbe56deb44bd'
python-version: '3.10'

# 2023-3.1
- conda_env_name: '2023-3.1-py310'
zenodo_id: '8342775'
md5_checksum: '06c16140f909b11ac6cb712c96c849af'
python-version: '3.10'

- conda_env_name: '2023-3.1-py310-tiled'
zenodo_id: '8342775'
md5_checksum: '1551a5591bfde30ddbb34d0798e4c94c'
python-version: '3.10'

# 2023-3.2
- conda_env_name: '2023-3.2-py310'
zenodo_id: '10056340'
md5_checksum: '71d187467850f10289ce0d8691aa80fd'
python-version: '3.10'

- conda_env_name: '2023-3.2-py310-tiled'
zenodo_id: '10056340'
md5_checksum: '8c3740791930b2f230e64b0f9399b7b8'
python-version: '3.10'
25 changes: 25 additions & 0 deletions .github/conda-env-tests/matrix_include.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/usr/bin/env python

"""matrix_include.py

Read a YAML configuration file and export the matrix.include field that is
used by GitHub Actions
"""

import os
import yaml
import sys


filename = os.getenv("MATRIX_INCLUDE_FILE", "")
if not filename:
print("You must set environment variable MATRIX_INCLUDE_FILE")
print("to the path of the YAML configuration file for matrix.include input.")
sys.exit(1)

with open(filename, "r") as config_file:
matrix_include = yaml.safe_load(config_file)

print(matrix_include)

sys.exit(0)
54 changes: 54 additions & 0 deletions .github/conda-env-tests/recent-envs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---

# 2023-3.3
- conda_env_name: '2023-3.3-py310'
zenodo_id: '10148425'
md5_checksum: 'bb77ea6f7a5bb63980d5181f7666ecc2'
python-version: '3.10'

- conda_env_name: '2023-3.3-py310-tiled'
zenodo_id: '10148425'
md5_checksum: '0a47934380db013b36f3e089afdff6aa'
python-version: '3.10'

- conda_env_name: '2023-3.3-py311'
zenodo_id: '10148425'
md5_checksum: 'dd6c06a02e43b286cdf84908d8339992'
python-version: '3.11'

- conda_env_name: '2023-3.3-py311-tiled'
zenodo_id: '10148425'
md5_checksum: '48d2a73ea743d34152400cd642fd7472'
python-version: '3.11'

# 2024-1.0
- conda_env_name: '2024-1.0-py310'
zenodo_id: '10548109'
md5_checksum: 'b6107d565f0ff9d7315eab96d28894e9'
python-version: '3.10'

- conda_env_name: '2024-1.0-py310-tiled'
zenodo_id: '10548109'
md5_checksum: '216c4df93c6893a7ebe6456b678ac89e'
python-version: '3.10'

- conda_env_name: '2024-1.0-py311'
zenodo_id: '10548109'
md5_checksum: '2592ed2002823c423b4a664183d384da'
python-version: '3.11'

- conda_env_name: '2024-1.0-py311-tiled'
zenodo_id: '10548109'
md5_checksum: 'a51aac76916f19f5982650c1c6f21503'
python-version: '3.11'

# 2024-2.0
- conda_env_name: '2024-2.0-py310-tiled'
zenodo_id: '11122851'
md5_checksum: '91955234f1617df158e7301f05a89b64'
python-version: '3.10'

- conda_env_name: '2024-2.0-py311-tiled'
zenodo_id: '11122851'
md5_checksum: 'd30ca6db3d93de99655a85a481c966c2'
python-version: '3.11'
52 changes: 52 additions & 0 deletions .github/workflows/_matrix_prep.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Prepare a matrix of NSLS-II Conda Environments

on:
workflow_call:

inputs:
config_prefix:
description: >
'YAML file prefix {value}-env.yaml for a configuration file that'
'conains the strategy.matrix.includes for conda environments'
required: true
type: string
default: recent

outputs:
matrix_include:
description: 'JSON array of strategy.matrix.include entries, as string'
value: ${{ jobs.matrix_prep.outputs.matrix_include }}

jobs:

matrix_prep:

runs-on: ubuntu-latest

outputs:
matrix_include: ${{ steps.set-matrix.outputs.matrix_include }}

steps:

- name: Check out the code repo
uses: actions/checkout@v4

- name: Install python interpreter
uses: actions/setup-python@v4
with:
python-version: "3.11"

- name: Install dependencies
run: |
set -vxeuo pipefail
python -m pip install pyyaml

- name: Import matrix.include from config file
id: set-matrix
run: |
set -vxeuo pipefail
export MATRIX_INCLUDE_DIR=${{ github.workspace }}/.github/conda-env-tests
export MATRIX_INCLUDE_FILE=${MATRIX_INCLUDE_DIR}/${{ inputs.config_prefix }}-envs.yml
echo "Building matrix from file ${MATRIX_INCLUDE_FILE}"
export MATRIX_INCLUDE_SCRIPT=${MATRIX_INCLUDE_DIR}/matrix_include.py
echo "matrix_include=$(python ${MATRIX_INCLUDE_SCRIPT})" >> $GITHUB_OUTPUT
Loading