-
-
Notifications
You must be signed in to change notification settings - Fork 353
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
93a8da3
commit 82e077b
Showing
6 changed files
with
197 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# Distributing Conda Package via conda-forge | ||
|
||
🚧 This page is under construction 🚧 | ||
|
||
<https://github.com/conda-forge/cantera-feedstock> |
158 changes: 158 additions & 0 deletions
158
doc/sphinx/develop/distribution-packages/pypi-sdist-wheel.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,158 @@ | ||
# Distributing sdist and Wheel Packages via PyPI | ||
|
||
Cantera has been [distributed on PyPI](https://pypi.org/project/Cantera/) as a source | ||
distribution (sdist) and a wheel since v2.6.0 in 2022. Wheels are Python's binary | ||
distribution format, where source code is compiled to platform-specific libraries to | ||
eliminate the need for user machines to have compilers and dependencies installed. | ||
Source distributions are also uploaded so that platforms where a wheel is not available | ||
can still attempt to build Cantera from the source code. | ||
|
||
This page describes how to build sdists and wheels for Cantera, and how these are built | ||
for distribution via PyPI. Some of the trade-offs in selecting the current build system | ||
are documented in the relevant [enhancement | ||
issue](https://github.com/Cantera/enhancements/issues/205). The source code to build and | ||
publish the wheels is in a [specialized repository on | ||
GitHub](https://github.com/Cantera/pypi-packages). | ||
|
||
The overall process is conducted in two steps: | ||
|
||
1. SCons is used to build an sdist from the git checkout with `scons sdist` | ||
2. cibuildwheel/`build` is used to build wheels from the sdist, **not the git checkout** | ||
|
||
There are a few reasons for this two-stage process. First, it ensures that our sdist can | ||
be used to build a wheel on users' systems. Second, it tremendously simplifies the | ||
cibuildwheel config, because installation of build dependencies is handled by | ||
cibuildwheel at build time. | ||
|
||
## TLDR | ||
|
||
This set of steps should build an sdist and wheel on a developer's machine for local | ||
testing. | ||
|
||
1. Clone the repository (submodules are optional) | ||
1. Install SCons and [build](https://pypi.org/project/build/) | ||
1. Run `scons sdist` | ||
a. `scons clean sdist` can also be used to automatically remove the | ||
`build` folder prior to building the `sdist. | ||
1. Building the wheel requires that Boost and libhdf5 are available on the build system. | ||
If these are not installed in standard locations, you can set `Boost_ROOT` and | ||
`HDF5_ROOT` environment variables to point to the correct location. | ||
1. Change to the `build/python_sdist/dist` folder | ||
1. Unpack the sdist: `tar xf cantera-3.1.0a2.tar.gz`. Note the version `3.1.0a2` may be | ||
different. | ||
1. Change into the unpacked directory | ||
1. Build the wheel: `python -m build --wheel .` Note the trailing `.` which indicates | ||
the current directory. You can also add the `-v` or `-vv` flags to increase build | ||
verbosity. | ||
|
||
The wheel should be located in the `build/python_sdist/dist/cantera-3.1.0a2/dist` | ||
folder. You can inspect the contents using any Zip-file reader. The CLI tool `unzip` can | ||
list the contents with the `-l` flag. You can install the wheel by running `python -m | ||
pip install <path to the wheel>`. | ||
|
||
## Source Distributions | ||
|
||
Sdists are built from the git checkout using SCons. Starting with v3.1, the only | ||
dependencies required to build the sdist are SCons and | ||
build. Dependencies such as NumPy, Boost, or any of | ||
the dependencies available as submodules are not necessary to build the sdist. | ||
|
||
To build the sdist, run `scons sdist` from the repository root. | ||
|
||
```{note} | ||
For versions 2.6 and 3.0, all the usual dependencies to build Cantera from source are required. Starting with 3.1, the build process for the sdist was simplified to pull in external dependencies at wheel-build time, rather than copying code from the Cantera external source tree into the sdist. | ||
``` | ||
|
||
The code for the sdist is located in the `interfaces/python_sdist` folder. Building the | ||
sdist takes two steps: | ||
|
||
1. SCons runs the `build_sdist.py` file to copy all the relevant source files into the | ||
`build/python_sdist` folder. | ||
2. SCons runs `build` to turn the source files into a compliant sdist tarball using the | ||
`scikit-build-core` package as the build backend. | ||
|
||
Both of these steps are done at "configure" time in SCons, when SCons is first parsing | ||
the SConstruct and SConscript files. If the build is successful, SCons will exit before | ||
performing any configuration or platform checks. This is why the build is done at | ||
configure time, to avoid these time-consuming checks that aren't necessary since the | ||
sdist just copies files into the `build` folder. | ||
|
||
If successful, the `.tar.gz` file will be located in the `build/python_sdist/dist` | ||
folder. | ||
|
||
### Sdist configuration | ||
|
||
The configuration for the sdist is done in the `pyproject.toml.in` file. This is a | ||
template file that is filled in at build time by the `build_sdist.py` script. It needs | ||
to be filled in by SCons because that's where we define the version of Cantera, in the | ||
`SConstruct` file. | ||
|
||
You can read more about the standard fields in the `pyproject.toml` file in the [PyPA | ||
documentation](https://packaging.python.org/en/latest/specifications/pyproject-toml/). | ||
You can read more about the `scikit-build` fields in the [`scikit-build-core` | ||
documentation](https://scikit-build-core.readthedocs.io/en/latest/). | ||
|
||
## Wheels | ||
|
||
As mentioned above, wheels (`.whl` files) are Python's standard binary distribution | ||
artifact. Similar to Conda packages, wheels include compiled libraries specific to a | ||
particular platform, so that end users don't need to have compilers installed to use a | ||
package. | ||
|
||
Building Cantera's wheels is done using `build`, similar to the sdist. However, the | ||
wheel is built from the unpacked sdist tarball, and you must pass the `--wheel` flag to | ||
`build`. Then, `build` will install `scikit-build-core` to manage building the wheel | ||
from Cantera's C++ and Cython source files. | ||
|
||
`scikit-build-core` uses CMake to manage the compiling and linking process for the | ||
wheel. Most of the configuration for the wheel build is therefore done in the | ||
`CMakeLists.txt` files in the `interfaces/python_sdist` folder. These are copied into | ||
the `build/python_sdist` folder and from there into the actual sdist tarball. | ||
|
||
```{note} | ||
The minimum required version of CMake is 3.24 to support all the options we use with | ||
`FetchContent`. A compatible version should be installed by scikit-build-core if it | ||
isn't available on the system. | ||
``` | ||
|
||
There are essentially 4 stages to the wheel build: | ||
|
||
1. External dependencies (`eigen`, `fmt`, `SUNDIALS`, `HighFive`, `yaml-cpp`) are either | ||
located on the system, or downloaded and compiled using the `FetchContent` mechanism | ||
1. Cantera's C++ source code is built and linked to the appropriate dependencies | ||
1. Cython runs and generates C++ source code for the Python interface | ||
1. The generated C++ is compiled, linked to `libcantera`, and packaged into the `.whl` | ||
file | ||
|
||
```{note} | ||
Cantera's other dependencies, specifically Boost and libhdf5, are neither installed nor | ||
built during the wheel build and must be pre-installed on the build system. I couldn't | ||
find a way to have eigen and HighFive, respectively, find those dependencies if they | ||
were also installed using `FetchContent`. If Boost and libhdf5 are installed in | ||
non-standard directories, you can use the `Boost_ROOT` or `Boost_INCLUDE_DIRS` and | ||
`HDF5_ROOT` variables to specific the correct locations. See the CMake documentation for | ||
`find_package()`. | ||
``` | ||
|
||
Steps 1 and 2 above are set up in the `interfaces/python_sdist/src/CMakeLists.txt` file. | ||
Steps 3 and 4 are set up in the `interfaces/python_sdist/cantera/CMakeLists.txt` file. | ||
|
||
## Publishing the sdist and Wheels | ||
|
||
When a version of Cantera is ready to be published to PyPI, the workflow in the | ||
[pypi-packages](https://github.com/Cantera/pypi-packages) repository must be executed. | ||
The workflow essentially conducts the steps in the [TLDR](#tldr) section above. | ||
|
||
The one big addition to the GitHub Action workflow is that there are a couple of shell | ||
scripts in the `pypi-packages` repository to build libhdf5 for macOS and Windows. On | ||
macOS, we also link to `libaec` so there's a small patch for the CMakeLists.txt file in | ||
`libaec`. | ||
|
||
For Linux builds, we have a [custom-built manylinux | ||
image](https://github.com/Cantera/hdf5-boost-manylinux) that already has the | ||
dependencies installed. This saves time compiling libhdf5 on Linux/ARM builds because | ||
those are emulated and already very slow. | ||
|
||
As much of the configuration for `cibuildwheel` as possible is done statically in | ||
`pyproject.toml.in`. Only values that are dynamically determined when the workflow runs | ||
should be set in the workflow. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# Distributing for Ubuntu via a Personal Package Archive (PPA) | ||
|
||
🚧 This page is under construction 🚧 | ||
|
||
<https://github.com/Cantera/cantera-ubuntu> |
7 changes: 7 additions & 0 deletions
7
doc/sphinx/develop/distribution-packages/windows-and-macos.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
# Distributing for Windows and macOS | ||
|
||
```{caution} | ||
These repositories are deprecated as of Cantera 3.1 with the removal of the legacy Matlab interface. When a new Matlab interface is provided, these repositories may be revived. | ||
``` | ||
|
||
🚧 This page is under construction 🚧 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters