Skip to content

Commit

Permalink
feat: support for osx-arm64 builds (#46775)
Browse files Browse the repository at this point in the history
* example recipe for osx-arm64

* test arm builds on macOS-14

* add arch flag

* try setting a higher sdk version

* check sdk versions

* set sdk earlier

* move logging

* set deployment target

* move var setting

* how old can we go?

* try 11.0 sdk

* trigger on PR

* Apply suggestions from code review

Co-authored-by: Martin Grigorov <[email protected]>

* zip a level up to match other ci

* must include noarch

* remove sdk workaround

* log deployment target

* use master cache

* case for job name

* initial merge wf (set up for testing)

* updates from testing

* source common.sh instead of directly setting env vars

* enable osx-arm64 on CircleCI

* xcode version (match github actions)

* debugging

* no docker for osx

* match bulk setup

* comment out run_conda_forge_build_setup

* Little more generic test

* longer build test

* clean up

* remove setup

* simplify and add osx-arm64

* revert

* try to match other CI

* standardize CI configurations for additional platforms

* fix scripts path

* fix check

* test skip

* reenable test packages

* add osx-arm64 to bulk

* test with user friendly name

* move arm jobs to existing GHA config

* test run_conda_forge_build_setup

* test run_conda_forge_build_setup on GitHub Actions

* debug sdk location

* try permissions

* run_conda_forge_build_setup is working on GHA

* clean up

* Try newer xcode version

* try x flag to see how noisy it is

* bump builds

* Apply suggestions from code review

This does add a little clutter from conda activate, but it could be useful for debugging some of the osx-specific stuff.

Co-authored-by: Martin Grigorov <[email protected]>

---------

Co-authored-by: Martin Grigorov <[email protected]>
  • Loading branch information
aliciaaevans and martin-g authored Jul 10, 2024
1 parent 04c45ac commit 21fbd60
Show file tree
Hide file tree
Showing 7 changed files with 294 additions and 106 deletions.
88 changes: 63 additions & 25 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
version: 2.1

executors:
# osx-arm64:
# macos:
# xcode: 14.2.0 # indicate your selected version of Xcode
# resource_class: macos.m1.large.gen1
osx-arm64:
macos:
xcode: 15.4.0
resource_class: macos.m1.medium.gen1
linux-aarch64:
machine:
image: ubuntu-2204:current
Expand All @@ -22,7 +22,13 @@ jobs: # a basic unit of work in a run

- run:
name: Check for Additional Platforms
command: ./.circleci/check-for-additional-platforms.sh "origin/master...HEAD" "build_and_test"
command: |
result=$(./scripts/check-for-additional-platforms.sh "origin/master...HEAD" "build_and_test" "${CIRCLE_JOB}")
if [[ ${result} != "build" ]]
then
echo "No recipes using this platform, skipping rest of job."
circleci-agent step halt
fi
- run:
name: Fetch bioconda install script
Expand All @@ -47,14 +53,29 @@ jobs: # a basic unit of work in a run
- run:
name: Build and test
command: |
. common.sh
source /opt/mambaforge/etc/profile.d/conda.sh
source /opt/mambaforge/etc/profile.d/mamba.sh
mamba activate bioconda
bioconda-utils build recipes config.yml \
--lint --docker --mulled-test \
--docker-base-image "quay.io/bioconda/bioconda-utils-build-env-cos7-$(arch):${BIOCONDA_UTILS_TAG#v}" \
--git-range origin/master HEAD
set -xe
eval "$(conda shell.bash hook)"
conda activate bioconda
source common.sh
# Sets up OSX SDK
if [[ $(uname) = "Darwin" ]]; then
if [[ "${OSX_SDK_DIR:-}" == "" ]]; then
OSX_SDK_DIR="$(xcode-select -p)/Platforms/MacOSX.platform/Developer/SDKs"
fi
sudo chmod o+rwx ${OSX_SDK_DIR}
run_conda_forge_build_setup
fi
if command -V docker >/dev/null; then
bioconda-utils build recipes config.yml \
--lint --docker --mulled-test \
--docker-base-image "quay.io/bioconda/bioconda-utils-build-env-cos7-$(arch):${BIOCONDA_UTILS_TAG#v}" \
--git-range origin/master HEAD
else
bioconda-utils build recipes config.yml \
--lint --git-range origin/master HEAD
fi
- run:
name: Prepare artifacts
Expand All @@ -63,7 +84,7 @@ jobs: # a basic unit of work in a run
mkdir -p /tmp/artifacts/packages
cd /opt/mambaforge/envs/bioconda/conda-bld || exit 0
find -name .cache | xargs rm -rf || true
for n in index.html channeldata.json linux-aarch64 linux-64 osx-64 noarch; do
for n in index.html channeldata.json linux-aarch64 linux-64 osx-64 osx-arm64 noarch; do
cp -rv $n /tmp/artifacts/packages || true
done
if command -V docker >/dev/null; then
Expand Down Expand Up @@ -93,7 +114,13 @@ jobs: # a basic unit of work in a run

- run:
name: Check for Additional Platforms
command: ./.circleci/check-for-additional-platforms.sh "${CIRCLE_SHA1}~1 ${CIRCLE_SHA1}" "build_and_upload"
command: |
result=$(./scripts/check-for-additional-platforms.sh "${CIRCLE_SHA1}~1 ${CIRCLE_SHA1}" "build_and_upload" "${CIRCLE_JOB}")
if [[ ${result} != "build" ]]
then
echo "No recipes using this platform, skipping rest of job."
circleci-agent step halt
fi
- run:
name: Fetch bioconda install script
Expand All @@ -118,9 +145,20 @@ jobs: # a basic unit of work in a run
- run:
name: Build and push
command: |
source /opt/mambaforge/etc/profile.d/conda.sh
source /opt/mambaforge/etc/profile.d/mamba.sh
mamba activate bioconda
set -xe
eval "$(conda shell.bash hook)"
conda activate bioconda
source common.sh
# Sets up OSX SDK
if [[ $(uname) = "Darwin" ]]; then
if [[ "${OSX_SDK_DIR:-}" == "" ]]; then
OSX_SDK_DIR="$(xcode-select -p)/Platforms/MacOSX.platform/Developer/SDKs"
fi
sudo chmod o+rwx ${OSX_SDK_DIR}
run_conda_forge_build_setup
fi
bioconda-utils handle-merged-pr recipes config.yml \
--repo bioconda/bioconda-recipes \
--git-range ${CIRCLE_SHA1}~1 ${CIRCLE_SHA1} \
Expand Down Expand Up @@ -184,10 +222,10 @@ jobs: # a basic unit of work in a run
name: Build and upload
command: |
set -e
. common.sh
source /opt/mambaforge/etc/profile.d/conda.sh
source /opt/mambaforge/etc/profile.d/mamba.sh
mamba activate bioconda
eval "$(conda shell.bash hook)"
conda activate bioconda
source common.sh
echo '============'
conda info --all
conda config --show-sources
Expand All @@ -214,7 +252,7 @@ workflows:
matrix:
parameters:
os:
#- osx-arm64
- osx-arm64
- linux-aarch64

build and upload (ARM):
Expand All @@ -226,7 +264,7 @@ workflows:
matrix:
parameters:
os:
#- osx-arm64
- osx-arm64
- linux-aarch64


Expand All @@ -239,6 +277,6 @@ workflows:
matrix:
parameters:
os:
#- osx-arm64
# - osx-arm64 Bulk is on GitHub Actions
- linux-aarch64
runner: [0, 1, 2, 3, 4, 5]
64 changes: 63 additions & 1 deletion .github/workflows/Bulk.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ jobs:
conda clean -y --all
build-osx:
name: Bulk OSX Builds
name: Bulk OSX-64 Builds
if: "contains(github.event.head_commit.message, '[ci run]')"
runs-on: macos-13
strategy:
Expand Down Expand Up @@ -121,3 +121,65 @@ jobs:
--lint --anaconda-upload --record-build-failures --skiplist-leafs \
--exclude bioconda-repodata-patches
conda clean -y --all
build-osx-arm:
name: Bulk OSX-ARM64 Builds
if: "contains(github.event.head_commit.message, '[ci run]')"
runs-on: macOS-14 # M1
strategy:
fail-fast: false
max-parallel: 4
matrix:
runner: [0, 1, 2, 3]
steps:
- uses: actions/checkout@v4
with:
# checkout as BiocondaBot in order to have the permission to push fail logs
token: ${{secrets.BIOCONDA_BOT_REPO_TOKEN}}

- name: set git user
run: |
git config user.name BiocondaBot
git config user.email [email protected]
- name: set path
run: echo "/opt/mambaforge/bin" >> $GITHUB_PATH

- name: Fetch conda install script
run: |
wget https://raw.githubusercontent.com/bioconda/bioconda-common/bulk/{common,install-and-set-up-conda,configure-conda}.sh
- name: Set up bioconda-utils
run: bash install-and-set-up-conda.sh

- name: Configure conda
run: bash configure-conda.sh

- name: Build and upload
env:
ANACONDA_TOKEN: ${{ secrets.ANACONDA_TOKEN }}
INVOLUCRO_AUTH: ${{ secrets.INVOLUCRO_AUTH }}
QUAY_OAUTH_TOKEN: ${{ secrets.QUAY_OAUTH_TOKEN }}
# Mimic circleci
OSTYPE: "darwin"
CI: "true"
run: |
set -xe
eval "$(conda shell.bash hook)"
conda activate bioconda
source common.sh
# Sets up OSX SDK
run_conda_forge_build_setup
echo '============'
conda info --all
conda config --show-sources
conda activate bioconda
python -c 'import bioconda_utils.utils as u ; import pathlib as p ; print(*(f"{f}:\n{p.Path(f).read_text()}" for f in u.load_conda_build_config().exclusive_config_files), sep="\n")'
echo '============'
bioconda-utils build recipes config.yml \
--worker-offset ${{ matrix.runner }} --n-workers 4 \
--lint --anaconda-upload --record-build-failures --skiplist-leafs \
--exclude bioconda-repodata-patches
conda clean -y --all
Loading

0 comments on commit 21fbd60

Please sign in to comment.