Add support for pyproject.toml

[agriyakhetarpal]

Description

There are a few nice-to-haves and caveats to this migration to pyproject.toml:

1. The setup.py file cannot be removed completely due to its flexibility with custom wheel build processes and steps to build extension modules, so this is more of an addition/conjugation with setuptools.build_meta rather than a complete migration—the latter is not possible without the inclusion of a new build-backend such as meson-build or scikit-build-core. Note: the scikit-build-core's support for editable installations is not mature and might break things, and scikit-build is the predecessor to it with limited support for editable installations, which is a deal-breaker.
2. Project metadata can now be included inside pyproject.toml, which simplifies the setuptools.setup() command. The CMakeBuild.py script is directly included inside setup.py to expose the linking of the IDAKLU extension and usage of the CMake arguments.

A list of tasks:

• Editable wheels on Windows, Linux, and macOS are working on GitHub Actions, and on ARM-based macOS machines locally. I will test them on a Windows system locally as well.
• pip is able to build non-editable wheels
• Both isolated and non-isolated builds work
• Unit and integration tests are passing on all platforms
• Doctests, example notebooks, and example scripts are passing
• cibuildwheel jobs pass for building reproducible wheels for all platforms

Fixes #3049 and supersedes #3053

Type of change

Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

• New feature (non-breaking change which adds functionality)
• Optimization (back-end change that speeds up the code)
• Bug fix (non-breaking change which fixes an issue)

Key checklist:

• No style issues: $ pre-commit run (or $ nox -s pre-commit) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
• All tests pass: $ python run-tests.py --all (or $ nox -s tests)
• The documentation builds: $ python run-tests.py --doctest (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ python run-tests.py --quick (or $ nox -s quick).

Further checks:

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[agriyakhetarpal]

So far unit and integration tests passed for both Windows and macOS but cibuildwheel failed on macOS and passed on Windows. With nox in the CI we run on a global installation of PyBaMM and not in a virtual environment, so we technically don't need to build without isolation. I am unsure how Read the Docs succeeds—we install the [docs] and [all] extras on an Ubuntu distribution, so it should have failed there like it did for the unit tests and other test groups on Ubuntu...

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (d5dae10) 99.58% compared to head (f1fd05f) 99.58%.
Report is 7 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3301   +/-   ##
========================================
  Coverage    99.58%   99.58%           
========================================
  Files          256      256           
  Lines        20119    20119           
========================================
  Hits         20036    20036           
  Misses          83       83           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Saransh-cpp]

I thought we were merging develop into main and adding a git tag on a commit on that branch to create a release.

The v23.9 branch will be merged into main to create the final release. Merging PRs in develop will not affect the final release in any way. Any bug/security fix that is required in the final release should be merged into develop (like other PRs) and then cherry-picked to the release branch (v23.9).

Of course, building wheels is not a security issue or a bug that needs fixing but rather a packaging feature

Yes, that sounds like a feature. It would be better to include that in the next release!

[agriyakhetarpal]

Makes sense, thanks for the explanation!

[arjxn-py]

I've tested wheels for amd64 Linux (WSL) and amd64 (Windows), they're being installed nicely. I've verified the solvers too by pybamm.have_idaklu() and all and also ran unit tests locally.
Overall tests are running as expected & those with solver dependencies are being skipped.

[Saransh-cpp]

@agriyakhetarpal could you merge develop? I was waiting for the release, but now that it is out, we should merge this as soon as possible

[agriyakhetarpal]

@agriyakhetarpal could you merge develop? I was waiting for the release, but now that it is out, we should merge this as soon as possible

Done. Since I am targeting some installation issues, improvements, and documentation in the coming time—it should be fine to merge once @kratman can review (a gentle reminder!)—I can add a minimal test to check for the IDAKLU solver during the cibuildwheel steps, but I think it would be better to do that in a follow-up PR since this is already very big.

[agriyakhetarpal]

I have disabled the wheel building on PRs like it always was and have triggered a final run on my fork, here is the link: https://github.com/agriyakhetarpal/PyBaMM/actions/runs/6935629522

[kratman]

Minor fix for the version.

Nox installs/text/examples work on my Apple M2. I will download the wheels and test them as well.

[agriyakhetarpal]

I will download the wheels and test them as well.

@arjxn-py has previously tested ans confirmed on WSL and Windows (amd64). We don't have arm64 wheels yet, so another test on a native Linux machine should be great and would suffice for the most part

[kratman]

I will download the wheels and test them as well.

@arjxn-py has previously tested ans confirmed on WSL and Windows (amd64). We don't have arm64 wheels yet, so another test on a native Linux machine should be great and would suffice for the most part

I am testing Linux Mint with x86_64 right now. Should be able to let you know if it works in a couple minutes

[kratman]

Works on my Linux machine

[Saransh-cpp]

Merging this, I hope everything works fine!

[agriyakhetarpal]

So glad to see the purple UI elements now, thanks a ton to everyone who helped me out!