Merge pull request #114 from ukhsa-collaboration/dev

Release 0.1.8

.github/pull_request_template.md

@@ -7,46 +7,28 @@ Replace this text with a description of the pull request including the issue num
 
 This is an itemised checklist for the QA process within UKHSA and represents the bare minimum a QA should be. 
 
-Full instructions on reviewing work can be found at Confluence on the [ONS QA of code guidance](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) page.
-
 **To the reviewer:** Check the boxes once you have completed the checks below.
 
-- [ ] It runs
-  - Can you get the code run to completion in a new instance and from top to bottom in the case of notebooks, or in a new R session?
-  - Can original analysis results be accurately & easily reproduced from the code?
-- [] tests pass
-- [] CI is successful
+- [ ] CI is successful
+  - Did the test suite run sucessfully?
+
 This is a basic form of Smoke Testing
 - [ ]  Data and security
-  - Use nbstripout to prevent Jupyter notebook output being committed to git repositories
-  - Files containing individual user's secret files and config files are not in repo, however examples of these files and setup instructions are included in the repo.
-  - Secrets include s3 bucket names, login credentials, and organisation information. These can be handled using secrets.yml
-  - If you are unsure whether an item should be secret please discuss with repo owner
-  - The changes do not include unreleased policy or official information.
+  - Files containing individual user's secret files and config files are not in repo.
+  - No private or identifiable data has been added to the repo.
+
 - [ ] Sensible
-  - Does the code execute the task accurately?  This is a subjective challenge.
+  - Does the code execute the task accurately? 
+  - Is the code tidy, commented and parsimonious?
   - Does the code do what the comments and readme say it does\*?
-  - Is the code robust enough to handle missing or challenging data?
+  - Is the code covered by useful unit tests?
+
 - [ ] Documentation
-  - The purpose of the code is clearly defined, whether in a markdown chunk at the top of a notebook or in a README
-  - Assumptions of the analysis & input data are clearly displayed to the reader, whether in a markdown chunk at the top of a notebook or in a README
+  - The purpose of the code is clearly defined?
+  - If reasonable, has an exaple of the code been given in a notebook in the docs?
   - Comments are included in the code so the reader can follow why the code behaves in the way it does
-  - Teams with high quality documentation are better able to implement technical practices more readily and perform better as a whole (DORA, 2021).
-  - Is the code written in a standard way? (In a hurry this may be a nice to have, but if at all possible, this standard from the beginning can cut long term costs dramatically)
-  - Code is modular, storing functions & classes in the src  and being imported into a notebook or script
-  - Projects should be based on the UKHSA repo template developed to work with cookiecutter
-  - Variable, function & module names should be intuitive to the reader
-    - For example, intuitive names include df_geo_lookup   & non-intuitive names include foobar
-  - Common and useful checks for coding we use broadly across UKHSA include:
-    - Rstyler
-    - lintr
-    - black
-    - flake8
-- [ ] Pair coding review completed (optional, but highly recommended for QA in a hurry)
-  - Pair programming is a way of working and reviewing that can result in the same standard of work being completed 40%-50% faster (Williams et al., 2000, Nosek, 1998) and is better than solo programming for tasks involving efficient knowledge transferring and for working on highly connected systems (Dawande et al., 2008).
-  - Have the assignee and reviewer been on a video call or in person together during the code development in a line by line writing and review process?
-
-\* If the comments or readme do not have enough information, this check fails.
+  - Is the code written in a standard way (does it pass linting)? 
+  - Variable, function & module names should be intuitive to the reader?
 
 ## How to QA this PR
 

.github/workflows/book.yml

@@ -30,23 +30,21 @@ jobs:
               uses: ts-graphviz/setup-graphviz@v1
 
             # install python
-            - name: Set up Python 3.8
+            - name: Set up Python 3.10
               uses: actions/setup-python@v4
               with:
-                python-version: 3.8
+                python-version: '3.10'
 
-            # install dependencies
-            - name: Install dependencies
+            # set up pygom
+            - name: Build and install pygom
               run: |
                 python -m pip install --upgrade pip
-                pip install -r requirements.txt
-                pip install -r docs/requirements.txt
+                pip install .
 
-            # set up pygom
-            - name: Build and install pygom
+            # install dependencies
+            - name: Install documentation dependencies
               run: |
-                  python setup.py build
-                  python setup.py install
+                pip install -r docs/requirements.txt
 
             # build the book
             # TODO check which flags are needed, -W
@@ -56,8 +54,16 @@ jobs:
 
             # deploy book to github-pages
             - name: GitHub Pages 
-              uses: peaceiris/[email protected]
+              uses: peaceiris/actions-gh-pages@v4
+              if: github.ref == 'refs/heads/main'
               with:
                 github_token: ${{ secrets.GITHUB_TOKEN }}
                 publish_dir: docs/_build/html
-
+            # deploy book to github-pages dev
+            - name: GitHub Pages dev 
+              uses: peaceiris/actions-gh-pages@v4
+              if: github.ref == 'refs/heads/dev'
+              with:
+                github_token: ${{ secrets.GITHUB_TOKEN }}
+                publish_dir: docs/_build/html
+                destination_dir: dev

.github/workflows/distribute_package.yml

@@ -0,0 +1,156 @@
+name: create PyGOM distributions
+
+on:
+  push:
+    branches:
+      - master
+      - dev
+      - feature/*
+      - bugfix/*
+
+  pull_request:
+    branches:
+      - master
+      - dev
+
+jobs:
+  build_wheels:
+    name: Build wheels on ${{ matrix.platform_id }} for Python v${{ matrix.python[1] }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      # Ensure that a wheel builder finishes even if another fails
+      fail-fast: false
+      matrix:
+        include:
+          # Window 64 bit
+          - os: windows-latest
+            python: [cp39, "3.9"]
+            platform_id: win_amd64
+          - os: windows-latest
+            python: [cp310, "3.10"]
+            platform_id: win_amd64
+          - os: windows-latest
+            python: [cp311, "3.11"]
+            platform_id: win_amd64
+          - os: windows-latest
+            python: [cp312, "3.12"]
+            platform_id: win_amd64
+
+          # Python 3.9 in the manylinux build environment requires our dependencies to be
+          # built from source so we won't supply a wheel for 3.9 (source build will prevent lib
+          # version conflicts).
+
+          # NumPy on Python 3.10 only supports 64bit and is only available with manylinux2014
+          - os: ubuntu-latest
+            python: [cp310, "3.10"]
+            platform_id: manylinux_x86_64
+            manylinux_image: manylinux2014
+          - os: ubuntu-latest
+            python: [cp311, "3.11"]
+            platform_id: manylinux_x86_64
+            manylinux_image: manylinux2014
+          - os: ubuntu-latest
+            python: [cp312, "3.12"]
+            platform_id: manylinux_x86_64
+            manylinux_image: manylinux2014
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # We need quite a deep fetch so that we get the versioning right 
+          fetch-depth: 500
+          fetch-tags: true
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      # Used to host cibuildwheel
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python[1] }}
+
+      - name: Install cibuildwheel
+        run: python -m pip install cibuildwheel>=2.19.2
+
+      # Need to duplicate these two lines as it seems Windows and Linux echo treats qutotes differently
+      - name: Build and test the wheels
+        if: matrix.os == 'ubuntu-latest'
+        run: python -m cibuildwheel --output-dir wheelhouse
+        env:
+          CIBW_BUILD: ${{ matrix.python[0] }}-${{ matrix.platform_id }}*
+          CIBW_TEST_COMMAND: python -W default -m unittest discover --start-directory {project}/tests
+          # setuptools_scm workaround for https://github.com/pypa/setuptools_scm/issues/455
+          CIBW_BEFORE_BUILD: ${{ github.ref == 'refs/heads/dev' &&  'echo ''local_scheme = "no-local-version"'' >> pyproject.toml && git diff --color=always && git update-index --assume-unchanged pyproject.toml' || '' }}
+      - name: Build and test the wheels
+        if: matrix.os == 'windows-latest'
+        run: python -m cibuildwheel --output-dir wheelhouse
+        env:
+          CIBW_BUILD: ${{ matrix.python[0] }}-${{ matrix.platform_id }}*
+          CIBW_TEST_COMMAND: python -W default -m unittest discover --start-directory {project}/tests
+          # setuptools_scm workaround for https://github.com/pypa/setuptools_scm/issues/455
+          CIBW_BEFORE_BUILD: ${{ github.ref == 'refs/heads/dev' &&  'echo local_scheme = "no-local-version" >> pyproject.toml && git diff --color=always && git update-index --assume-unchanged pyproject.toml' || '' }}
+
+      # Upload the results
+      - uses: actions/upload-artifact@v4
+        with:
+          name: cibw-wheels-${{ matrix.platform_id }}-${{ matrix.python[0] }}
+          path: ./wheelhouse/*.whl
+
+  build_sdist:
+    name: Build source distribution
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # We need quite a deep fetch so that we get the versioning right 
+          fetch-depth: 500
+          fetch-tags: true
+
+      # setuptools_scm workaround for https://github.com/pypa/setuptools_scm/issues/455
+      - name: Disable local version identifier on develop CI
+        if: github.ref == 'refs/heads/dev'
+        run: |
+          echo 'local_scheme = "no-local-version"' >> pyproject.toml
+          git diff --color=always
+          git update-index --assume-unchanged pyproject.toml
+
+      - name: Build sdist
+        run: pipx run build --sdist
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: cibw-sdist
+          path: dist/*.tar.gz
+
+  upload_pypi:
+    name: Upload release to PyPI
+    needs: [build_wheels, build_sdist]
+    runs-on: ubuntu-latest
+    environment: 
+      name: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          # unpacks all CIBW artifacts into dist/
+          pattern: cibw-*
+          path: dist
+          merge-multiple: true
+
+      # This is the live push to PyPI on tagging the master branch
+      - uses: pypa/gh-action-pypi-publish@release/v1
+        # Upload to PyPI on every tag starting with 'v' 
+        if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
+        with:
+          repository-url: https://pypi.org/p/pygom
+
+      # Upload to PyPI *testing* every dev branch commit 
+      - uses: pypa/gh-action-pypi-publish@release/v1
+        # Upload to testing only if we are on the dev branch
+        if: github.ref == 'refs/heads/dev'
+        with:
+          # Testing only at this point
+          repository-url: https://test.pypi.org/legacy/