Build wheels for more platforms (Linux aarch64 wheels, MUSL wheels, etc.)

[agriyakhetarpal]

Description

Currently, we build wheels for amd64 (Linux, macOS, Windows) and also pypy wheels on Windows (https://pypi.org/project/pybamm/#files). However, we don't provide wheels for arm64 platforms yet neither for Linux nor M-series macOS.

This means that users installing from PyPI use the source distributions, which do not contain compiled extensions. Their compilation requires pybind11 and the presence of SUNDIALS, SuiteSparse, and CasADi to link against which is easier done with an installation from source (not available and accessible for all users since this needs elevated or administrator permissions on the host system). Otherwise, pip falls back to building and storing a wheel without a platform and an ABI tag (none-any.whl) on the machine since the IDAKLU component is optional by default and doesn't compile if CMake or pybind11 are not found. This might be (slightly) easier to do after #3049 is completed.

Motivation

Easier and streamlined installation on the aforementioned platforms since the compiled extension is bundled into the wheels along with its shared external libraries (they are repaired to recreate static linkage and their runtime search paths altered)

Possible Implementation

GitHub Actions has some larger-sized M-series macOS runners now with smaller runners coming in the future. Linux arm64 runners are also on their roadmap. We can however

1. use emulation with either QEMU (recommended by the Scientific Python guides) or Dockcross (casadi) to build arm64 wheels
2. the Apple Xcode tools are also capable of cross-compilation to build macOS arm64 wheels in the form of a universal wheel which is double the size

Additional context

https://cibuildwheel.readthedocs.io/en/stable/faq/

Task list

• macOS arm64 wheels (Use self-hosted macOS M2 runner to build macOS arm64 or universal wheels #3772, Build M-series (arm64) macOS wheels on GitHub Actions hosted runners #3789)
• Linux aarch64 wheels
• MUSL wheels (if needed – note: might not work due to CasADi)

[agriyakhetarpal]

The macOS arm64 builds were successful here: https://github.com/agriyakhetarpal/PyBaMM/actions/runs/6599848898/job/17929588884#step:10:1796 through cross-compilation. I am now trying to fix troubles with not being able to find BLAS on Linux arm64. I guess openblas-devel isn't available in the index, so I have tried linking with blas-devel currently...

[agriyakhetarpal]

I was able to make some progress on this and get a BLAS distribution working using the EPEL repository for yum on CentOS 7 (on aarch64 using QEMU) on this branch. SUNDIALS can link to OpenBLAS successfully and the build-time requirements pass; however, there is a different issue now: casadi cannot be imported on pypa/manylinux2014 containers running on aarch64 architectures and returns this error during our CMake build process (we currently try importing casadi by executing the Python interpreter to get its path where it exists in the site-packages directory):

ImportError: /opt/python/cp38-cp38/lib/python3.8/site-packages/casadi/libcasadi.so.3.7:

undefined symbol: _ZTINSt6thread6_StateE

Full stack trace here: https://github.com/agriyakhetarpal/PyBaMM/actions/runs/6605214684/job/17940157625

and thus the idaklu target cannot be linked to it and compilation cannot proceed further. It is difficult to find what function or object this mangled C++ symbol represents, running objdump -T on the libcasadi.so.3.7 shared object file to analyse C++ library calls returns UND, unlike the rest of the output for the symbols.

This issue looks to be related: casadi/casadi#2887. From the discussion, it seems like it has to do with a discrepancy between the versions of the C++ compiler or a difference between the compilers that were used themselves. We would need to either use the workaround in the issue thread (I'm not sure whether it will work or whether it is even recommended for use in shared libraries since it might cause the IDAKLU extension to be non-importable with Python, even) or wait for an upstream fix. The workaround, if it should be used, could be something like:

if(CMAKE_CXX_COMPILER_ID STREQUAL "some older compiler version")
    target_link_libraries(idaklu ${SUNDIALS_LIBRARIES} casadi PRIVATE -Wl,--unresolved-symbols=ignore-in-shared-libs)
endif()

(Note: I am not super proficient with CMake so the implementation could turn out to be a bit different)

I assume this issue is the reason casadi does not use the cibuildwheel pipelines and instead uses a custom Docker image and dockcross to cross-compile for this platform. For now, I have opened #3465 to support the creation of macOS arm64 wheels which have been cross-compiled using Xcode. We can keep this issue open for a while if we wish to distribute Linux arm64 wheels in future, or close it with the linked pull request if we don't want to support this platform. In any case, installing PyBaMM from source on modern arm64 systems or via our multi-platform Docker images (where the casadi import and hence linkage works) is much easier now (cc: @martinjrobins @Saransh-cpp)

P.S. It could also be an issue with emulation using QEMU but is less likely because I can reproduce the failure in a Docker container on an M-series macOS machine. It is however possible to write another job in the workflow file that would use dockcross to do this, but it would be some work.