Merge pull request #28 from wellcomecollection/rk/gha-build

Move build to GitHub Actions

.buildkite/pipeline.yml

@@ -1,13 +1,3 @@
 steps:
-  - command: .buildkite/scripts/convert_notebooks.sh
-    if: build.branch != "main"
-    label: "convert notebooks from .ipynb to .md"
-
-  - command: .buildkite/scripts/build_html.sh
-    if: build.branch != "main"
-    label: "build HTML"
-
-  - commands:
-      - .buildkite/scripts/deploy_to_github.sh
-    if: build.branch == "main"
-    label: "deploy changes to GitHub"
+  - command: true
+    label: "Take no action"

.github/workflows/deploy.yml

@@ -0,0 +1,58 @@
+name: "Deploy to GitHub Pages"
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  build:
+    name: Build documentation
+    permissions:
+        id-token: write
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Convert Notebooks
+      run: scripts/convert_notebooks.sh
+    - name: Commit update
+      run: |
+        git config --global user.name 'WeCo bot'
+        git config --global user.email '[email protected]'
+        git commit -am "Update notebooks" || exit 0
+        git push
+    - name: Install Node
+    # https://github.com/actions/setup-node?tab=readme-ov-file#usage
+    # The node-version input is optional. If not supplied, the node version from PATH will be used.
+    # However, it is recommended to always specify Node.js version and don't rely on the system one.
+      uses: actions/setup-node@v4
+      with:
+        node-version: 20
+    - name: Install dependencies
+      run: yarn install --frozen-lockfile
+    - name: Build website
+      run: yarn build
+    - name: Upload Build Artifact
+      uses: actions/upload-pages-artifact@v3
+      with:
+        path: build
+
+
+  deploy:
+    name: Deploy to GitHub Pages
+    needs: build
+
+    # Grant GITHUB_TOKEN the permissions required to make a Pages deployment
+    permissions:
+      pages: write # to deploy to Pages
+      id-token: write # to verify the deployment originates from an appropriate source
+
+    # Deploy to the github-pages environment
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+
+    runs-on: ubuntu-latest
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

.github/workflows/test.yml

@@ -0,0 +1,36 @@
+name: "Test deployment"
+on:
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  build:
+    name: Build documentation
+    permissions:
+        id-token: write
+        contents: write
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        ref: ${{ github.event.pull_request.head.ref }}
+    - name: Convert Notebooks
+      run: scripts/convert_notebooks.sh
+    - name: Commit update
+      run: |
+        git config --global user.name 'WeCo bot'
+        git config --global user.email '[email protected]'
+        git commit -am "Update notebooks" || exit 0
+        git push
+    - name: Install Node
+    # https://github.com/actions/setup-node?tab=readme-ov-file#usage
+    # The node-version input is optional. If not supplied, the node version from PATH will be used.
+    # However, it is recommended to always specify Node.js version and don't rely on the system one.
+      uses: actions/setup-node@v4
+      with:
+        node-version: 20
+    - name: Install dependencies
+      run: yarn install --frozen-lockfile
+    - name: Build website
+      run: yarn build

docs/examples/01-exploring-wellcome-collections-apis.md

@@ -1,6 +1,6 @@
 # 1. Exploring Wellcome Collection's APIs
 
-[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/01-exploring-wellcome-collections-apis.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/01-exploring-wellcome-collections-apis.ipynb)
+[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/01-exploring-wellcome-collections-apis.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/01-exploring-wellcome-collections-apis.ipynb)
 
 Wellcome collection has a few public APIs which can be used to fetch things like works, images, and concepts. They all live behind the following base URL
 

docs/examples/02-extracting-more-data-for-local-analysis.md

@@ -1,6 +1,6 @@
 # 2. Extracting more data for local analysis
 
-[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/02-extracting-more-data-for-local-analysis.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/02-extracting-more-data-for-local-analysis.ipynb)
+[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/02-extracting-more-data-for-local-analysis.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/02-extracting-more-data-for-local-analysis.ipynb)
 
 In the last notebook, we saw that the `/works` API can do some clever querying and filtering. However, we often have questions which can't be answered by the API by itself. In those cases, it's useful to collect a load of data from the API and then analyse it locally.
 

docs/examples/03-connecting-the-apis-together.md

@@ -1,6 +1,6 @@
 # 3. Connecting the APIs together
 
-[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/03-connecting-the-apis-together.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/03-connecting-the-apis-together.ipynb)
+[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/03-connecting-the-apis-together.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/03-connecting-the-apis-together.ipynb)
 
 So far, we've only looked at the `/works` API, but Wellcome Collection has a few more which we can make use of. As well as `/works`, we can also use `/images` and `/concepts`. 
 

docs/examples/04-building-graphs-of-visually-similar-images.md

@@ -1,6 +1,6 @@
 # 4. Building graphs of visually similar images
 
-[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/04-building-graphs-of-visually-similar-images.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/04-building-graphs-of-visually-similar-images.ipynb)
+[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/04-building-graphs-of-visually-similar-images.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/04-building-graphs-of-visually-similar-images.ipynb)
 
 In the last notebook, we introduced the ability to fetch visually similar images using the `/images` API.
 

docs/examples/05-working-with-snapshots-of-the-api.md

@@ -1,6 +1,6 @@
 # 5. Working with snapshots of the API
 
-[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/05-working-with-snapshots-of-the-api.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/05-working-with-snapshots-of-the-api.ipynb)
+[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/05-working-with-snapshots-of-the-api.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/05-working-with-snapshots-of-the-api.ipynb)
 
 As we saw at the end of the last notebook, the API limits its responses to 10,000 total results - after that point, users are directed to work with the snapshots. For example, making a request to <https://api.wellcomecollection.org/catalogue/v2/works?pageSize=100&page=101> gives us:
 

docs/examples/06-visualising-the-collection-on-a-map.md

@@ -1,6 +1,6 @@
 # 6. Visualising the collections on a map
 
-[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/06-visualising-the-collection-on-a-map.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/06-visualising-the-collection-on-a-map.ipynb)
+[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/06-visualising-the-collection-on-a-map.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/06-visualising-the-collection-on-a-map.ipynb)
 
 In this notebook, we're going to use a secondary API to visualise the geographical extent of the collection on a map. 
 

docs/examples/07-building-an-image-classifier.md

@@ -1,6 +1,6 @@
 # 7. Building an image classifier
 
-[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/07-building-an-image-classifier.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/07-building-an-image-classifier.ipynb)
+[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/07-building-an-image-classifier.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/07-building-an-image-classifier.ipynb)
 
 This notebook is going to race through some high-level concepts in machine learning (specifically, fine-tuning a convolutional neural network). However, our focus will remain on demonstrating the practical uses of the Wellcome Collection API. As such, some important ML topics will be covered in less detail than they deserve, and some will be skipped entirely.
 If you're not already familiar with the basics of ML but want to learn more, I'd recommend exploring [Practical Deep Learning for Coders](https://course.fast.ai/) by fast.ai. It describes itself as:

docs/examples/08-extracting-features-from-text.md

@@ -1,6 +1,6 @@
 # 8. Extracting features from text
 
-[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/08-extracting-features-from-text.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/08-extracting-features-from-text.ipynb)
+[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/08-extracting-features-from-text.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/08-extracting-features-from-text.ipynb)
 
 In the last notebook, we saw that using a pre-trained network allowed us to extract features from images, and train a classifier for new categories on top of those features. We can do the same thing with text, using a pre-trained network to extract features from text. In this notebook, we'll use those features the visualise the similarities and differences between works in the collection, and try to find clusters of related material.
 

docs/examples/09-building-a-recommender-system-for-subjects.md

@@ -1,6 +1,6 @@
 # 9. Building a recommender system for subjects
 
-[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/09-building-a-recommender-system-for-subjects.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/tree/HEAD/notebooks/09-building-a-recommender-system-for-subjects.ipynb)
+[View on GitHub](https://github.com/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/09-building-a-recommender-system-for-subjects.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/wellcomecollection/developers.wellcomecollection.org/blob/rk/gha-build/notebooks/09-building-a-recommender-system-for-subjects.ipynb)
 
 Finally, we'll consider building a recommender system using data from Wellcome Collection. Thes machine learning models work slightly differently to the ones we've seen so far. Rather than being trained to predict a single value, they're trained to predict a whole matrix of interactions between two kinds of entity.
 

.buildkite/scripts/convert_notebooks.sh → scripts/convert_notebooks.sh

@@ -27,7 +27,7 @@ docker run --rm --tty \
     jupyter/scipy-notebook \
     jupyter nbconvert \
     --to markdown \
-    --template .buildkite/scripts/mdoutput \
+    --template "$root/scripts/mdoutput" \
     --output-dir "$root/docs/examples" \
     $root/notebooks/*.ipynb
 
@@ -67,14 +67,13 @@ for file in $root/docs/examples/*.md; do
   fi
 done
 
-
 # add a link to github and colab for each notebook
 GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
 
 for file in $root/notebooks/*.ipynb; do
   filename=$(basename -- "$file")
 
-  github_path="wellcomecollection/developers.wellcomecollection.org/tree/$GIT_BRANCH/notebooks/$filename"
+  github_path="wellcomecollection/developers.wellcomecollection.org/blob/$GIT_BRANCH/notebooks/$filename"
   github_url="https://github.com/$github_path"
   colab_url="https://colab.research.google.com/github/$github_path"
 
@@ -83,22 +82,3 @@ for file in $root/notebooks/*.ipynb; do
   # insert the line at the second line of the file
   awk -v line="$line" 'NR==2{print line}1' "$path" > tmp && mv -f tmp "$path"
 done
-
-# commit any changes back to the branch
-if [[ `git status --porcelain` ]]; then
-  git config user.name "Buildkite on behalf of Wellcome Collection"
-  git config user.email "[email protected]"
-
-  git remote add ssh-origin $BUILDKITE_REPO || true
-  git fetch ssh-origin
-  git checkout --track ssh-origin/$BUILDKITE_BRANCH || true
-
-  git add --verbose --update
-  git commit -m "Convert notebooks"
-
-  git push ssh-origin HEAD:$BUILDKITE_BRANCH
-  exit 1;
-else
-  echo "No changes from notebook conversion"
-  exit 0;
-fi