Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a CI job that builds a Docker multi-arch manifest list #249

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add a CI job that builds a Docker multi-arch manifest list #249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 84 additions & 1 deletion .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ on:
- cron: "0 6 * * SUN" # Once weekly on Sunday @ 0600 UTC
workflow_dispatch: null

env:
REGISTRY_DH: docker.io/condaforge
REGISTRY_QUAY: quay.io/condaforge

jobs:
build:
name: ${{ matrix.cfg.DOCKERIMAGE }}:${{ matrix.cfg.DOCKERTAG }} (${{ matrix.cfg.DISTRO_ARCH }})
Expand Down Expand Up @@ -253,17 +257,96 @@ jobs:
--build-arg DISTRO_NAME \
--build-arg DISTRO_VER \
-t condaforge/$DOCKERIMAGE:$DOCKERTAG \
-t ${{ env.REGISTRY_DH }}/$DOCKERIMAGE:$DOCKERTAG \
-t ${{ env.REGISTRY_QUAY }}/$DOCKERIMAGE:$DOCKERTAG \
-f ${DOCKERFILE:-${DOCKERIMAGE}}/Dockerfile \
--no-cache --squash .

- name: Run image
run: |
./.circleci/run_docker_build.sh

- name: Deploy
- name: Deploy images
if: github.ref == 'refs/heads/main' && github.repository == 'conda-forge/docker-images'
env:
CFD_QUAY_USER: conda_forge_daemon
CFD_QUAY_PASSWORD: ${{ secrets.CFD_QUAY_PASSWORD }}
DH_USER: condaforgebot
DH_PASSWORD: ${{ secrets.DH_PASSWORD }}
run: |
./scripts/deploy

build-manifest:
needs: [build]
if: github.ref == 'refs/heads/main' && github.repository == 'conda-forge/docker-images'
name: ${{ matrix.docker-registry }}/${{ matrix.cfg.DOCKER_MANIFEST }}:${{ matrix.cfg.DOCKER_TAG }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
docker-registry: [docker.io/condaforge, quay.io/condaforge]
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not possible to use the env vars here (REGISTRY_DH and REGISTRY_QUAY) :-/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this help?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately this also does not work!
commit: martin-g@426b612
error: https://github.com/martin-g/docker-images/actions/runs/7474578018
Fails with:

Invalid workflow file: .github/workflows/ci.yaml#L291
You have an error in your yaml syntax on line 291

cfg:
- DOCKER_MANIFEST: linux-anvil-cos7
DOCKER_TAG: "latest"
DOCKER_IMAGES: "<<ORG>>/linux-anvil-cos7-x86_64:<<TAG>>,<<ORG>>/linux-anvil-ppc64le:<<TAG>>,<<ORG>>/linux-anvil-aarch64:<<TAG>>"
- DOCKER_MANIFEST: linux-anvil-alma
DOCKER_TAG: "8"
DOCKER_IMAGES: "<<ORG>>/linux-anvil-alma-x86_64:<<TAG>>,<<ORG>>/linux-anvil-alma-ppc64le:<<TAG>>,<<ORG>>/linux-anvil-alma-aarch64:<<TAG>>"
- DOCKER_MANIFEST: linux-anvil-ubi-cuda
DOCKER_TAG: "11.0"
DOCKER_IMAGES: "<<ORG>>/linux-anvil-cuda:<<TAG>>,<<ORG>>/linux-anvil-ppc64le-cuda:<<TAG>>,<<ORG>>/linux-anvil-aarch64-cuda:<<TAG>>"
- DOCKER_MANIFEST: linux-anvil-cos7-cuda
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is some repetition for all DOCKER_MANIFEST: linux-anvil-cos7-cuda but I wasn't able to compose a more compact matrix (e.g. by using DOCKER_TAG: ["11.1", "11.2", "11.3", ...]).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok. We already have some repetition for the CUDA images

Also longer term we are slowly phasing the CUDA images out entirely. It will take us some time to get there though

We could probably take this opportunity to drop CUDA 11.0 & 11.1 as we don't use those any more ( conda-forge/conda-forge-pinning-feedstock#1708 )

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you suggest to drop them in this PR or in another ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to do in this PR. Just if they are causing difficulty, we should feel comfortable dropping them

DOCKER_IMAGES: "<<ORG>>/linux-anvil-cuda:<<TAG>>,<<ORG>>/linux-anvil-ppc64le-cuda:<<TAG>>,<<ORG>>/linux-anvil-aarch64-cuda:<<TAG>>"
DOCKER_TAG: "11.1"
- DOCKER_MANIFEST: linux-anvil-cos7-cuda
DOCKER_IMAGES: "<<ORG>>/linux-anvil-cuda:<<TAG>>,<<ORG>>/linux-anvil-ppc64le-cuda:<<TAG>>,<<ORG>>/linux-anvil-aarch64-cuda:<<TAG>>"
DOCKER_TAG: "11.2"
- DOCKER_MANIFEST: linux-anvil-cos7-cuda
DOCKER_IMAGES: "<<ORG>>/linux-anvil-cuda:<<TAG>>,<<ORG>>/linux-anvil-ppc64le-cuda:<<TAG>>,<<ORG>>/linux-anvil-aarch64-cuda:<<TAG>>"
DOCKER_TAG: "11.3"
- DOCKER_MANIFEST: linux-anvil-cos7-cuda
DOCKER_IMAGES: "<<ORG>>/linux-anvil-cuda:<<TAG>>,<<ORG>>/linux-anvil-ppc64le-cuda:<<TAG>>,<<ORG>>/linux-anvil-aarch64-cuda:<<TAG>>"
DOCKER_TAG: "11.4"
- DOCKER_MANIFEST: linux-anvil-cos7-cuda
DOCKER_IMAGES: "<<ORG>>/linux-anvil-cuda:<<TAG>>,<<ORG>>/linux-anvil-ppc64le-cuda:<<TAG>>,<<ORG>>/linux-anvil-aarch64-cuda:<<TAG>>"
DOCKER_TAG: "11.5"
- DOCKER_MANIFEST: linux-anvil-cos7-cuda
DOCKER_IMAGES: "<<ORG>>/linux-anvil-cuda:<<TAG>>,<<ORG>>/linux-anvil-ppc64le-cuda:<<TAG>>,<<ORG>>/linux-anvil-aarch64-cuda:<<TAG>>"
DOCKER_TAG: "11.6"
- DOCKER_MANIFEST: linux-anvil-cos7-cuda
DOCKER_IMAGES: "<<ORG>>/linux-anvil-cuda:<<TAG>>,<<ORG>>/linux-anvil-ppc64le-cuda:<<TAG>>,<<ORG>>/linux-anvil-aarch64-cuda:<<TAG>>"
DOCKER_TAG: "11.7"
- DOCKER_MANIFEST: linux-anvil-cos7-cuda
DOCKER_IMAGES: "<<ORG>>/linux-anvil-cuda:<<TAG>>,<<ORG>>/linux-anvil-ppc64le-cuda:<<TAG>>,<<ORG>>/linux-anvil-aarch64-cuda:<<TAG>>"
DOCKER_TAG: "11.8"

steps:
- uses: actions/checkout@v4

- name: Interpolate placeholders
id: interpolate
run: |
set -x
INTERPOLATED=`echo "${{ matrix.cfg.DOCKER_IMAGES }}" | sed "s#<<ORG>>#${{ matrix.docker-registry }}#g" | sed "s#<<TAG>>#${{ matrix.cfg.DOCKER_TAG }}#g"`
echo "DOCKER_IMAGES=${INTERPOLATED}" >> "$GITHUB_OUTPUT"

- name: Login to DockerHub registry
uses: docker/login-action@v2
with:
username: condaforgebot
password: ${{ secrets.DH_PASSWORD }}

- name: Login to Quay.io registry
uses: docker/login-action@v2
with:
registry: quay.io
username: conda_forge_daemon
password: ${{ secrets.CFD_QUAY_PASSWORD }}

- name: Push Docker manifest list for ${{ matrix.docker-registry }}
uses: Noelware/[email protected]
with:
inputs: ${{ matrix.docker-registry }}/${{ matrix.cfg.DOCKER_MANIFEST }}:${{ matrix.cfg.DOCKER_TAG }}
images: ${{ steps.interpolate.outputs.DOCKER_IMAGES }}
push: true

9 changes: 4 additions & 5 deletions scripts/deploy
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
#!/bin/bash

# dockerhub
docker login -u condaforgebot -p $DH_PASSWORD
docker push condaforge/$DOCKERIMAGE:$DOCKERTAG
docker login -u ${DH_USER} -p ${DH_PASSWORD}
docker push ${REGISTRY_DH}/$DOCKERIMAGE:$DOCKERTAG

# quay.io
docker login -u conda_forge_daemon -p ${CFD_QUAY_PASSWORD} quay.io
docker tag condaforge/$DOCKERIMAGE:$DOCKERTAG quay.io/condaforge/$DOCKERIMAGE:$DOCKERTAG
docker push quay.io/condaforge/$DOCKERIMAGE:$DOCKERTAG
docker login -u ${CFD_QUAY_USER} -p ${CFD_QUAY_PASSWORD} quay.io
docker push ${REGISTRY_QUAY}/$DOCKERIMAGE:$DOCKERTAG