Skip to content

Commit

Permalink
Support CPython 3.11, 3.12, and aarch64 processors (#2331)
Browse files Browse the repository at this point in the history 
* fix[ml]: adjust tests to reflect latest apis of 3rd party libraries (xgboost, lightgbm)

* Build wheels on pull_request

* Maximize wheel build parallellization

* git submodule update --remote --merge packages/vaex-core/vendor/pybind11

* Fix gcc error

* Fix workflow syntax

* Remove redefinition

https://github.com/pybind/pybind11/blob/769fd3b889fef6cddb060f2a0be26aee62b4da05/include/pybind11/pytypes.h#L859

https://github.com/ddelange/vaex/actions/runs/3965609112/jobs/6795506653#step:6:2110

* Disable win32

https://github.com/ddelange/vaex/actions/runs/3965689146/jobs/6795667118#step:6:538

* Remove testing leftovers

* Add upper cap on lightgbm

microsoft/LightGBM#5196 (comment)

* Migrate to mamba-org/setup-micromamba@v1

* Upload release assets on Github (pre)release

* Add missing permission for release assets

* Build cp312 wheels

* Remove setuptools and wheel from pyproject.toml

* Replace imp with importlib

* chore: trigger ci

* ci: upgrade xcode for brew install libomp

* update requirements-ml to comply with the latest veex-ml expectations.

* try to install lightgbm via pip

* fix: only the mini taxi file is on s3 for cost savings

* ci: pin dask<2024.2.0 to get the same hash keys

This version gives different results, although
not a problem in production (it will make your
cache invalid though), for CI we test that we
have stable keys (fingerprints)

* fix: only the mini taxi file is on s3 for cost savings (2)

* ci: skip notebooks that depend on the dataframe server

* ci: skip ci steps in various old python versions

* Bump micromamba and other actions

* ci: specific test hangs on ci+osx

* Bump cibuildwheel, use native arm64 mac runners

* test: log is renamed to log loss

* test: skip lightgbm tests on 36 and 37

* test: skip sklearn tests on 36 and 37

* Update packages/vaex-core/setup.py

Co-authored-by: ddelange <[email protected]>

* chore: drop python 3.6 and 3.7 support

Co-authored-by: Ewout ter Hoeven <[email protected]>

* test: skip a failing test for windows

* ci: macOS 11 is retired as of June 28

* ci: always build wheels, but do not publish instead

* ci: try with older micromamba, 1.5.6 seems to sometimes hang

* Install setuptools in wheel.yml

* Add sudo

* Fix windows

* Use sudo only on macos

* Fix empty string evaluating false

* Add .readthedocs.yaml

* Pull submodules

* Try editable rtd install

* Try move editable install to requirements_rtd.txt

* Allow newer sphinx

* Autocancel previous runs in PRs

* Autocancel all jobs

* Sphinx sidebar fix

https://github.com/dmwyatt/sphinx-book-theme/blob/2416ef6cebc3d83019dbc83f25c61b72719cad55/docs/customize/sidebar-primary.md#default-sidebar-elements

* Remove autocancel, doesn't work from forks

* Add cancel-in-progress keys

* Amend to only cancel on PRs

* Add CIBW_TEST_COMMAND

* Disallow numpy v2

* Skip musllinux due to misding blake3 wheels

* Install carco for blake3 source install

* Fix CIBW_ENVIRONMENT

* Add CIBW_TEST_SKIP

* Build wheels with numpy v1

* run tests on 3.10-3.12

* upgrade micromamba

* unpin pytest-asyncio

* try different pin as <2.0 because mamba crashes on that

* Fix dephell and CI hell (ddelange#3)

* allow for multiple fingerprints related to package versions

* explicit dtype=object

* fix pandas issues

* revert graphene upgrade

* force uv to ignore Werkzeug (we do not use it, its a subdependency which has conflicts)

* fix: work with modern scipy/sparse arrays

* skip test for python 3.8

* skip test on windows

* add missing import

* skip some tests conditionally

* tuple comparison fix

* does osx run without sourcing .bash profile?

* skip a few more tests

* only run test on linux

* Try building arm64 wheels on macos-13

* Switch to older Xcode

* Fix readthedocs by manually unpinning werkzeug in the graphene-tornado v2.6.1 release

# for now we leave graphene-tornado / graphene on v2
# ref #2356 (comment)
# in top level vaex[ci] we added the following (additional) direct reference requirement which adds one commit (ddelange/graphene-tornado@d75f01f) on top of the 2.6.1 release to unpin werkzeug
# "graphene-tornado @ https://github.com/ddelange/graphene-tornado/archive/refs/heads/2.6.1.unpin-werkzeug.zip",

* Simplify

* Revert "Simplify"

This reverts commit 91263bc.

* Add back tags event trigger for core wheel upload

* Switch to trusted publishing

* Add trusted publishing for vaex-core

* Exclude windows 3.8 from pythonpackage.yml

* Don't upload vaex-core source distribution

* Disable cp38-win vaex-core wheel build

* Add permission

---------

Co-authored-by: Jovan Veljanoski <[email protected]>
Co-authored-by: Maarten A. Breddels <[email protected]>
Co-authored-by: Ewout ter Hoeven <[email protected]>
  • Loading branch information
@ddelange @JovanVeljanoski
@maartenbreddels @EwoutH
4 people authored Sep 11, 2024
1 parent 15245cf commit e3e1842
Show file tree
Hide file tree
Showing 90 changed files with 604 additions and 1,563 deletions.
233 changes: 55 additions & 178 deletions .github/workflows/pythonpackage.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,130 +5,74 @@ on:
branches:
- master
pull_request:
branches:
- master
- docs
schedule:
- cron: "0 2 * * 1-5"

# to make bash use ~/.bashrc, see https://github.com/marketplace/actions/setup-miniconda#example-5-custom-installer
concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

defaults:
run:
shell: bash -l {0}

concurrency:
group: ${{ github.ref }}
cancel-in-progress: true
shell: bash

jobs:
main:
env:
CACHE_NUMBER_COMPILE: 6
MAMBA_ROOT_PREFIX: /dev/shm/mambaroot
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
max-parallel: 12
matrix:
os: [ubuntu-latest, windows-latest, macOS-11]
python-version: [3.6, 3.7, 3.8, 3.9]
os: [ubuntu-latest, windows-latest, macos-13]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
exclude:
- {os: windows-latest, python-version: "3.8"}

steps:
- name: disk space
if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os == 'ubuntu-latest') }}
run: |
df -h
- uses: maxim-lobanov/setup-xcode@v1
# alternative would be to upgrade tapi to 1100.0.11, we can possibly remove this in the future
if: matrix.os == 'macOS-11'
with:
xcode-version: "11.7.0"
- uses: actions/checkout@v2
with:
submodules: true
- uses: actions/checkout@v2
if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os == 'ubuntu-latest') }}
with:
repository: vaexio/vaex-enterprise
token: ${{ secrets.PAT_PULL_ENTERPRISE }}
path: vaex-enterprise
- name: install micromamba
uses: mamba-org/provision-with-micromamba@main

- uses: actions/setup-python@v5
with:
environment-name: vaex-dev
environment-file: ci/conda-env.yml
extra-specs: |
python=${{ matrix.python-version }}
- name: Extra non-windows installs
python-version: ${{ matrix.python-version }}

- run: python -m pip install --upgrade pip setuptools wheel uv

# - uses: actions/checkout@v2
# if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os == 'ubuntu-latest') }}
# with:
# repository: vaexio/vaex-enterprise
# token: ${{ secrets.PAT_PULL_ENTERPRISE }}
# path: vaex-enterprise

- name: Setup Graphviz
uses: ts-graphviz/setup-graphviz@v2

- name: Setup PCRE (Nix-only)
if: matrix.os != 'windows-latest'
run: |
echo "not needed for now"
# - name: Extra windows installs
# if: matrix.os == 'windows-latest'
# run: |
# # not sure why, but the conda env is strange (old dask) if we put this constraint in the conda solver
# # also might solve scipy issues
# micromamba update scipy ipython dask
# micromamba install scipy=1.7.1
# - name: free disk space
# if: matrix.os == 'ubuntu-latest'
# run: |
# sudo swapoff -a
# sudo rm -f /swapfile
# sudo apt clean
# docker rmi $(docker image ls -aq)
# df -h
run: sudo -E bash bin/install_pcre.sh

- name: Install OpenMP runtime (Mac-only)
if: matrix.os == 'macOS-11'
run: |
brew install libomp
- name: Cache compiled binaries
# this fails for this combination, leading to binaries filled with 0's
if: matrix.python-version != '3.6' || matrix.os != 'macOS-11'
id: cache-compiled-binaries
uses: actions/cache@v2
with:
path: |
packages/vaex-core/build/lib*/vaex/vaexfast*
packages/vaex-core/build/lib*/vaex/super*
key: ${{ runner.OS }}-${{ matrix.python-version }}-${{ hashFiles('packages/vaex-core/src/*') }}-${{ env.CACHE_NUMBER_COMPILE }}
- name: Fix cache timestamp
run: |
touch packages/vaex-core/build/lib*/vaex/vaexfast* || echo "ok to fail"
touch packages/vaex-core/build/lib*/vaex/super* || echo "ok to fail"
ls -alh packages/vaex-core/build/lib*/vaex/ || echo "ok to fail"
if: startswith(matrix.os, 'macos')
run: brew install libomp

- name: Copy dll (Windows-only)
if: (matrix.os == 'windows-latest')
uses: ./ci/actions/windll

- name: Install vaex
run: |
pip install myst_parser
pip install -e . -v
# ./ci/03-install-vaex.sh micromamba
run: uv pip install -v --system .[ci]

- run: uv pip freeze

# - name: Install vaex-enterprise
# if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os == 'ubuntu-latest') }}
# run: |
# cd vaex-enterprise
# micromamba install -c conda-forge distributed
# pip install ray
# pip install -e .
- name: Copy dll
if: (matrix.os == 'windows-latest')
uses: ./ci/actions/windll
- name: Build Python packages
run: |
mkdir dist
(cd packages/vaex-core && python setup.py bdist_wheel); cp packages/vaex-core/dist/* dist
(cd packages/vaex-graphql && python setup.py bdist_wheel); cp packages/vaex-graphql/dist/* dist
(cd packages/vaex-jupyter && python setup.py bdist_wheel); cp packages/vaex-jupyter/dist/* dist
(cd packages/vaex-ml && python setup.py bdist_wheel); cp packages/vaex-ml/dist/* dist
(cd packages/vaex-viz && python setup.py bdist_wheel); cp packages/vaex-viz/dist/* dist
(cd packages/vaex-astro && python setup.py bdist_wheel); cp packages/vaex-astro/dist/* dist
(cd packages/vaex-hdf5 && python setup.py bdist_wheel); cp packages/vaex-hdf5/dist/* dist
(cd packages/vaex-server && python setup.py bdist_wheel); cp packages/vaex-server/dist/* dist
(cd packages/vaex-meta && python setup.py bdist_wheel); cp packages/vaex-server/dist/* dist
- name: Upload builds
uses: actions/upload-artifact@v2
with:
name: dist-${{ github.run_number }}-${{ matrix.python-version }}-${{ matrix.os }}
path: ./dist

# - name: Lint with flake8
# run: |
# echo "soon"
Expand All @@ -137,123 +81,56 @@ jobs:
# # flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
# # flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Install fixes for Python 3.6
if: matrix.python-version == '3.6'
run: |
# by default on 3.6 we get an old version, so manually upgrade
pip install gcsfs==0.8.0
- name: Install pytest-asyncio correct version
if: matrix.python-version != '3.10'
run: |
# 3.10 doesn't seem to work
pip install "pytest-asyncio<0.14"

- name: Test with pytest
run: |
./ci/04-run-test-suite.sh
- name: Test with pytest (with cache on)
if: matrix.python-version == '3.9' && matrix.os == 'ubuntu-latest'
if: matrix.python-version == '3.12' && matrix.os == 'ubuntu-latest'
run: |
VAEX_CACHE_RESULTS=1 ./ci/04-run-test-suite.sh
- name: Check ml spec
# no catboost for py39 and py37
if: matrix.python-version != '3.9' && matrix.os != 'windows-latest' && matrix.python-version != '3.7'
run: |
python -m vaex.ml.spec packages/vaex-ml/vaex/ml/spec_new.json
diff packages/vaex-ml/vaex/ml/spec_new.json packages/vaex-ml/vaex/ml/spec.json
# - name: Test with pytest (vaex-enterprise)
# if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os == 'ubuntu-latest') }}
# run: |
# py.test vaex-enterprise/tests --timeout=1000

- name: Test notebooks
if: matrix.os != 'windows-latest' && matrix.python-version != '3.6'
run: |
./ci/05-run-notebooks.sh
if: matrix.os != 'windows-latest'
run: ./ci/05-run-notebooks.sh

- name: Authenticate Google Cloud Platform
if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && !((matrix.os == 'windows-latest') || (matrix.os == 'macOS-11' && matrix.python-version == '3.6')) }}
if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && !((matrix.os == 'windows-latest')) }}
uses: google-github-actions/setup-gcloud@v0
with:
project_id: ${{ secrets.GCP_PROJECT_ID_VAEX }}
service_account_key: ${{ secrets.GCP_SA_KEY_VAEX }}
export_default_credentials: true

- name: Test vaex-contrib
# do not run in a PR from someone else, skip windows, and osx+py36
if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && !((matrix.os == 'windows-latest') || (matrix.os == 'macOS-11' && matrix.python-version == '3.6')) }}
# do not run in a PR from someone else, skip windows
if: ${{ (github.event.pull_request.head.repo.full_name == 'vaexio/vaex') && (matrix.os != 'windows-latest') }}
env:
PROJECT_ID: ${{ secrets.GCP_PROJECT_ID_VAEX }}
run: |
pip install -e 'packages/vaex-contrib[gcp]'
./ci/06-run-contrib-tests.sh
install:
runs-on: ${{ matrix.os }}
needs: [main]

strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macOS-11]
python-version: [3.7, 3.8, 3.9]
# ssl/certifi issues with this combination
exclude:
- os: windows-latest
python-version: 3.7

steps:
- uses: actions/checkout@v2
- uses: actions/download-artifact@v2
with:
name: dist-${{ github.run_number }}-${{ matrix.python-version }}-${{ matrix.os }}
path: ./dist

- name: Install micromamba
uses: mamba-org/provision-with-micromamba@main
with:
environment-name: vaex-test
environment-file: ci/conda-base-minimal.yml
extra-specs: |
python=${{ matrix.python-version }}
pcre
rich
# it shouldn't be needed to use pcre, maybe we should make the wheels similar to how we make distributions
- name: Cache pip
uses: actions/cache@v2
env:
# Increase this value to reset cache
CACHE_NUMBER: 0
with:
path: |
~/.cache/pip
key:
pip-${{ runner.os }}-${{ matrix.python-version }}-${{ env.CACHE_NUMBER }}

- name: Install core packages
run: |
pip install -vv dist/vaex*core* dist/vaex*hdf5*
- name: Test core import
run: |
python -c "import vaex; df = vaex.example()"
- name: Install all packages
run: |
pip install -vv dist/*
- name: Test import
run: |
python -c "import vaex; df = vaex.example()"
run: ./ci/06-run-contrib-tests.sh

- name: Test comand line
run: |
vaex convert ~/.vaex/data/helmi-dezeeuw-2000-FeH-v2-10percent.hdf5 test.parquet
pip install rich
VAEX_PROGRESS_TYPE=rich vaex convert ~/.vaex/data/helmi-dezeeuw-2000-FeH-v2-10percent.hdf5 test.parquet
vaex settings yaml
vaex settings md
- name: Test server
if: matrix.os != 'windows-latest'
run: |
vaex server --add-example --port 9999&
vaex server --add-example --port 9999 &
- name: Wait for Vaex server
if: matrix.os != 'windows-latest'
Expand Down
Loading

0 comments on commit e3e1842

Please sign in to comment.