Add support for multi-arch docker images #399

yuvipanda · 2022-10-19T02:59:08Z

Install appropriate version of mambaforge based on build architecture
Generate lockfiles on base-notebook and pangeo-notebook for:
- osx-64 (x86_64)
- osx-arm64 (Apple Sillicon M1/M2/M3 chips)
- linux-aarch64 (arm64)
- ~~ppc64le (POWER8/9)~~

github-actions · 2022-10-19T02:59:22Z

👈 Try on Mybinder.org!
👈 Try on Pangeo GCP Binder!
👈 Try on Pangeo AWS Binder!

The version of mambaforge appropriate for the architecture that the docker build is running on is built this way. Ref pangeo-data#396

yuvipanda · 2022-10-19T03:06:38Z

conda-lock also needs to know though.

yuvipanda · 2022-10-19T03:26:58Z

/condalock

scottyhq · 2022-10-19T17:28:34Z

.github/workflows/CondaLock.yml

+      - name: Specify which architectures to build lockfiles for
+        run: |
+          # space delimited list of architectures to send to conda-lock
+          echo "BUILD_ARCHS='64 aarch64 ppc64le'" > ${GITHUB_ENV}


i think it'd be best to just start with one additional? aarch64 and add others down the line if need be

@scottyhq yeah i agree I mostly added both to test what packages are needed to be fixed for ppc64le

Let's build just for linux-aarch64 first as mentioned at #396 (comment)

scottyhq · 2022-10-19T17:35:51Z

.github/workflows/CondaLock.yml

-          conda-lock lock --mamba -k explicit -f environment.yml -p linux-64
-          ../generate-packages-list.py conda-linux-64.lock > packages.txt
+          for ARCH in ${BUILD_ARCHS}; do
+            conda-lock lock --mamba -k explicit -f environment.yml -p linux-${ARCH}


this might be a good time to switch to a unified lockfile and actually add conda-lock to the base environment. (Our current usage of conda-lock pre-dates the unified lockfile format now shown here https://github.com/conda-incubator/conda-lock#basic-usage). Installing the environment with conda-lock would actually recognize the host platform and choose the correct set of packages:

conda-lock lock -f environment.yml -p linux-64 aarch64 conda-lock install -n pangeo-notebook

the drawback is that if someone just has conda or mamba installed they dont recognize the new unified lockfile format... so you could not do conda env create -f conda-lock.yml ...

@scottyhq ok that's pretty cool and we clearly should just move to that.

Thinking that we might need to update from conda-lock=1.4 to conda-lock=2.x first at

pangeo-docker-images/environment-condalock.yml

Line 6 in 554675c

- conda-lock=1.4

Also FYI, mamba doesn't support installing from the unified conda-lock.yml, but micromamba does according to https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html#conda-lock-yaml-spec-files. See also mamba-org/mamba#1884 (comment)

scottyhq · 2022-10-19T17:39:52Z

Makefile

+	conda-lock lock --mamba -k explicit -f environment.yml -p linux-aarch64; \
+	../generate-packages-list.py conda-linux-aarch64.lock > packages-aarch64.txt; \
+	conda-lock lock --mamba -k explicit -f environment.yml -p linux-ppc64le; \
+	../generate-packages-list.py conda-linux-ppc64le.lock > packages-ppc64le.txt; \


see comment above on unified lockfile. but i still really like a simple text file as well though! maybe we create a artifacts subdirectory to organize files not actually used by docker build?

scottyhq · 2022-10-19T17:43:28Z

Makefile

+	conda-lock lock --mamba -k explicit -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-aarch64; \
+	../generate-packages-list.py conda-linux-aarch64.lock > packages-aarch64.txt; \
+	conda-lock lock --mamba -k explicit -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-ppc64le; \
+	../generate-packages-list.py conda-linux-ppc64le.lock > packages-ppc64le.txt; \


TBH I have rarely used this Makefile and am always just building things via GitHub Actions ;) I've put in a little more time to embracing docker buildx and think it might be the right tool to evolve this repo. in particular the bake tool for several related images https://github.com/scottyhq/pangeo-buildx#usage

@scottyhq I completely agree - I love that repo and think that's what this should be :D What do you think needs to happen to complete that transition?

Awesome! Mainly just more time :) I abandoned it for a bit with the "dont fix what isn't broken" rationale :) but it's also nice to keep evolving so I'll try to open a PR here this week and would love your review

scottyhq · 2024-06-26T17:09:51Z

@yuvipanda @weiji14 I'm thinking we should go ahead and merge this. Likely there will be some things to work out, but linux and mac ARM support will be a nice addition

weiji14

@yuvipanda @weiji14 I'm thinking we should go ahead and merge this. Likely there will be some things to work out, but linux and mac ARM support will be a nice addition

Yep, agree with going ahead with this. Let me just test running conda-lock locally on some of these images to check that there are no serious issues. Also a few minor suggestions below 👇

base-image/Dockerfile

pangeo-notebook/environment.yml

weiji14

Ok, tested things locally, and locking on base-notebook and pangeo-notebook worked, but ml-notebook and pytorch-notebook didn't. Maybe we just do base-/pangeo- for now in this PR, and I can do a follow-up one for ml-/pytorch-?

.github/workflows/CondaLock.yml

weiji14 · 2024-06-26T23:28:32Z

.github/workflows/CondaLock.yml

+          cd ml-notebook
+          conda-lock lock -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-64 -p linux-aarch64 -p osx-64 -p osx-arm64


Tensorflow doesn't have a linux-aarch64 package yet - conda-forge/tensorflow-feedstock#136

Could not lock the environment for platform linux-aarch64 Could not solve for environment specs The following package could not be installed └─ tensorflow >=2.15.0 cuda120* does not exist (perhaps a typo or a missing channel).

There are osx-64/osx-arm64 packages on conda-forge, though only CPU builds. Not sure if the osx-arm64 build actually uses the Apple Sillicon GPU.

weiji14 · 2024-06-26T23:33:07Z

.github/workflows/CondaLock.yml

+          cd pytorch-notebook
+          conda-lock lock -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-64 -p linux-aarch64 -p osx-64 -p osx-arm64


Pytorch does have linux-aarch64/osx-64/osx-arm64 packages on conda-forge, but we set a cuda120 pin in the environment.yml file so locking fails:

Could not lock the environment for platform linux-aarch64 Could not solve for environment specs The following packages are incompatible ├─ pytorch >=2.1.0 cuda120* is not installable because it requires │ └─ __cuda, which is missing on the system; └─ torchvision >=0.16.1 cuda120* does not exist (perhaps a typo or a missing channel).

Co-authored-by: Wei Ji <[email protected]>

scottyhq · 2024-06-27T17:50:50Z

.github/workflows/Test.yml

+    - name: Set up QEMU
+      uses: docker/setup-qemu-action@v3


following https://docs.docker.com/build/ci/github-actions/multi-platform/.

Yep, will need to use virtualization, until someone can pay us to use native arm64 runners (https://github.blog/changelog/2024-06-24-github-actions-ubuntu-24-04-image-now-available-for-arm64-runners) 🙂

scottyhq · 2024-06-27T17:52:14Z

Added some commits to update the Makefile and Actions to actually build for multiple platforms, and it seems to be working!

weiji14

Pre-approving since this is mostly ready, but have a look at the suggestions first. We'll probably need to disable osx-64 for ml-notebook and pytorch-notebook because of the (Linux-only) CUDA pin set in the environment.yml file.

weiji14 · 2024-06-27T22:38:36Z

.github/workflows/Test.yml

+    - name: Set up QEMU
+      uses: docker/setup-qemu-action@v3


Yep, will need to use virtualization, until someone can pay us to use native arm64 runners (https://github.blog/changelog/2024-06-24-github-actions-ubuntu-24-04-image-now-available-for-arm64-runners) 🙂

.github/workflows/CondaLock.yml

Makefile

base-image/Dockerfile

Co-authored-by: Wei Ji <[email protected]>

yuvipanda mentioned this pull request Oct 19, 2022

Images availability for OS/ARCH other than linux/amd64 #396

Closed

yuvipanda requested a review from scottyhq October 19, 2022 03:00

Don't hardcode CPU architecture in dockerfile

1c576c6

The version of mambaforge appropriate for the architecture that the docker build is running on is built this way. Ref pangeo-data#396

yuvipanda force-pushed the uname branch from 665c1f8 to 1c576c6 Compare October 19, 2022 03:03

yuvipanda marked this pull request as draft October 19, 2022 03:24

Build conda lock files for amd64, aarch64 & ppcle64

d69c208

yuvipanda force-pushed the uname branch from b53c855 to d69c208 Compare October 19, 2022 03:25

pangeo-bot and others added 3 commits October 19, 2022 03:32

[condalock-command] autogenerated conda-lock files

fb94a8f

Dynamically pick conda-lock file based on architecture

b941a1d

Generate multi-arch lock files in Makefile

25cef13

yuvipanda changed the title ~~Don't hardcode CPU architecture in dockerfile~~ Add support for multi-arch docker images Oct 19, 2022

scottyhq reviewed Oct 19, 2022

View reviewed changes

weiji14 linked an issue Jun 27, 2023 that may be closed by this pull request

Images availability for OS/ARCH other than linux/amd64 #396

Closed

scottyhq and others added 3 commits June 4, 2024 12:34

merge main

5f34442

tidy dockerfile and make, comment out arm-incompatible packages

9d2802e

Merge branch 'master' into uname

0db80af

scottyhq marked this pull request as ready for review June 26, 2024 17:08

scottyhq requested a review from weiji14 June 26, 2024 17:08

weiji14 reviewed Jun 26, 2024

View reviewed changes

base-image/Dockerfile Outdated Show resolved Hide resolved

pangeo-notebook/environment.yml Outdated Show resolved Hide resolved

weiji14 reviewed Jun 26, 2024

View reviewed changes

scottyhq and others added 3 commits June 26, 2024 17:12

Update pangeo-notebook/environment.yml

4f25c69

Co-authored-by: Wei Ji <[email protected]>

Update .github/workflows/CondaLock.yml

0642772

Co-authored-by: Wei Ji <[email protected]>

miniforge3 not mambaforge

511947a

scottyhq added 2 commits June 26, 2024 17:47

multiarch cmds in makefile and github actions

ac916f0

Test multiarch

d96bc35

scottyhq reviewed Jun 27, 2024

View reviewed changes

weiji14 approved these changes Jun 27, 2024

View reviewed changes

Apply suggestions from code review

38c293e

Co-authored-by: Wei Ji <[email protected]>

scottyhq merged commit fc92d8b into pangeo-data:master Jun 28, 2024
5 checks passed

This was referenced Jun 28, 2024

Set dockerfile syntax version #559

Merged

Relock for ARM64 #561

Merged

Revert to conda-lock matrix job #562

Merged

This was referenced Jun 29, 2024

Pin jaxlib to use cuda120 build with cuda-nvcc and update README note #549

Merged

Publish CalVer tags for linux/arm64 platform builds #564

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for multi-arch docker images #399

Add support for multi-arch docker images #399

yuvipanda commented Oct 19, 2022 •

edited by weiji14

Loading

github-actions bot commented Oct 19, 2022

yuvipanda commented Oct 19, 2022

yuvipanda commented Oct 19, 2022

scottyhq Oct 19, 2022

yuvipanda Oct 19, 2022

weiji14 Jul 20, 2023

scottyhq Oct 19, 2022

yuvipanda Oct 19, 2022

weiji14 Jul 20, 2023 •

edited

Loading

scottyhq Oct 19, 2022

scottyhq Oct 19, 2022

yuvipanda Oct 19, 2022

scottyhq Oct 19, 2022

scottyhq commented Jun 26, 2024

weiji14 left a comment

weiji14 left a comment •

edited

Loading

weiji14 Jun 26, 2024 •

edited

Loading

weiji14 Jun 26, 2024

scottyhq Jun 27, 2024

weiji14 Jun 27, 2024

scottyhq commented Jun 27, 2024

weiji14 left a comment

weiji14 Jun 27, 2024

		cd ml-notebook
		conda-lock lock -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-64 -p linux-aarch64 -p osx-64 -p osx-arm64

		cd pytorch-notebook
		conda-lock lock -f environment.yml -f ../pangeo-notebook/environment.yml -f ../base-notebook/environment.yml -p linux-64 -p linux-aarch64 -p osx-64 -p osx-arm64

Add support for multi-arch docker images #399

Add support for multi-arch docker images #399

Conversation

yuvipanda commented Oct 19, 2022 • edited by weiji14 Loading

github-actions bot commented Oct 19, 2022

yuvipanda commented Oct 19, 2022

yuvipanda commented Oct 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weiji14 Jul 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scottyhq commented Jun 26, 2024

weiji14 left a comment

Choose a reason for hiding this comment

weiji14 left a comment • edited Loading

Choose a reason for hiding this comment

weiji14 Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scottyhq commented Jun 27, 2024

weiji14 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuvipanda commented Oct 19, 2022 •

edited by weiji14

Loading

weiji14 Jul 20, 2023 •

edited

Loading

weiji14 left a comment •

edited

Loading

weiji14 Jun 26, 2024 •

edited

Loading