Merge pull request pybamm-team#3344 from arjxn-py/nbmake

Revamped notebook testing infrastructure

.gitignore

@@ -8,6 +8,7 @@
 *.mat
 *.csv
 *.hidden
+*.pkl
 
 # don't ignore important .txt and .csv files
 !requirements*

CONTRIBUTING.md

@@ -120,7 +120,9 @@ This allows people to (1) use PyBaMM without ever importing Matplotlib and (2) c
 
 All code requires testing. We use the [unittest](https://docs.python.org/3.3/library/unittest.html) package for our tests. (These tests typically just check that the code runs without error, and so, are more _debugging_ than _testing_ in a strict sense. Nevertheless, they are very useful to have!)
 
-If you have nox installed, to run unit tests, type
+We also use [pytest](https://docs.pytest.org/en/latest/) along with the [nbmake](https://github.com/treebeardtech/nbmake) and the [pytest-xdist](https://pypi.org/project/pytest-xdist/) plugins to test the example notebooks.
+
+If you have `nox` installed, to run unit tests, type
 
 ```bash
 nox -s unit
@@ -151,26 +153,48 @@ When you commit anything to PyBaMM, these checks will also be run automatically
 
 ### Testing the example notebooks
 
-To test all the example notebooks in the `docs/source/examples/` folder, type
+To test all the example notebooks in the `docs/source/examples/` folder with `pytest` and `nbmake`, type
 
 ```bash
 nox -s examples
 ```
 
-To edit the structure and how the Jupyter notebooks get rendered in the Sphinx documentation (using `nbsphinx`), install [Pandoc](https://pandoc.org/installing.html) on your system, either using `conda` (through the `conda-forge` channel)
+Alternatively, you may use `pytest` directly with the `--nbmake` flag:
 
 ```bash
-conda install -c conda-forge pandoc
+pytest --nbmake
 ```
 
-or refer to the [Pandoc installation instructions](https://pandoc.org/installing.html) specific to your platform.
+which runs all the notebooks in the `docs/source/examples/notebooks/` folder in parallel by default, using the `pytest-xdist` plugin.
+
+Sometimes, debugging a notebook can be a hassle. To run a single notebook, pass the path to it to `pytest`:
+
+```bash
+pytest --nbmake docs/source/examples/notebooks/notebook-name.ipynb
+```
+
+or, alternatively, you can use posargs to pass the path to the notebook to `nox`. For example:
+
+```bash
+nox -s examples -- docs/source/examples/notebooks/notebook-name.ipynb
+```
+
+You may also test multiple notebooks this way. Passing the path to a folder will run all the notebooks in that folder:
 
-If notebooks fail because of changes to PyBaMM, it can be a bit of a hassle to debug. In these cases, you can create a temporary export of a notebook's Python content using
+```bash
+nox -s examples -- docs/source/examples/notebooks/models/
+```
+
+You may also use an appropriate [glob pattern](https://www.malikbrowne.com/blog/a-beginners-guide-glob-patterns) to run all notebooks matching a particular folder or name pattern.
+
+To edit the structure and how the Jupyter notebooks get rendered in the Sphinx documentation (using `nbsphinx`), install [Pandoc](https://pandoc.org/installing.html) on your system, either using `conda` (through the `conda-forge` channel)
 
 ```bash
-python run-tests.py --debook docs/source/examples/notebooks/notebook-name.ipynb script.py
+conda install -c conda-forge pandoc
 ```
 
+or refer to the [Pandoc installation instructions](https://pandoc.org/installing.html) specific to your platform.
+
 ### Testing the example scripts
 
 To test all the example scripts in the `examples/` folder, type
@@ -242,6 +266,7 @@ This also means that, if you can't fix the bug yourself, it will be much easier
       ```
 
       This will start the debugger at the point where the `ValueError` was raised, and allow you to investigate further. Sometimes, it is more informative to put the try-except block further up the call stack than exactly where the error is raised.
+
    2. Warnings. If functions are raising warnings instead of errors, it can be hard to pinpoint where this is coming from. Here, you can use the `warnings` module to convert warnings to errors:
 
       ```python
@@ -251,15 +276,19 @@ This also means that, if you can't fix the bug yourself, it will be much easier
       ```
 
       Then you can use a try-except block, as in a., but with, for example, `RuntimeWarning` instead of `ValueError`.
+
    3. Stepping through the expression tree. Most calls in PyBaMM are operations on [expression trees](https://github.com/pybamm-team/PyBaMM/blob/develop/docs/source/examples/notebooks/expression_tree/expression-tree.ipynb). To view an expression tree in ipython, you can use the `render` command:
 
       ```python
       expression_tree.render()
       ```
 
       You can then step through the expression tree, using the `children` attribute, to pinpoint exactly where a bug is coming from. For example, if `expression_tree.jac(y)` is failing, you can check `expression_tree.children[0].jac(y)`, then `expression_tree.children[0].children[0].jac(y)`, etc.
+
 3. To isolate whether a bug is in a model, its Jacobian or its simplified version, you can set the `use_jacobian` and/or `use_simplify` attributes of the model to `False` (they are both `True` by default for most models).
+
 4. If a model isn't giving the answer you expect, you can try comparing it to other models. For example, you can investigate parameter limits in which two models should give the same answer by setting some parameters to be small or zero. The `StandardOutputComparison` class can be used to compare some standard outputs from battery models.
+
 5. To get more information about what is going on under the hood, and hence understand what is causing the bug, you can set the [logging](https://realpython.com/python-logging/) level to `DEBUG` by adding the following line to your test or script:
 
    ```python3
@@ -330,7 +359,7 @@ And then visit the webpage served at http://127.0.0.1:8000. Each time a change t
 
 Major PyBaMM features are showcased in [Jupyter notebooks](https://jupyter.org/) stored in the [docs/source/examples directory](docs/source/examples/notebooks). Which features are "major" is of course wholly subjective, so please discuss on GitHub first!
 
-All example notebooks should be listed in [docs/sourceexamples/index.rst](https://github.com/pybamm-team/PyBaMM/blob/develop/docs/source/examples/index.rst). Please follow the (naming and writing) style of existing notebooks where possible.
+All example notebooks should be listed in [docs/source/examples/index.rst](https://github.com/pybamm-team/PyBaMM/blob/develop/docs/source/examples/index.rst). Please follow the (naming and writing) style of existing notebooks where possible.
 
 All the notebooks are tested daily.
 

docs/source/examples/notebooks/expression_tree/expression-tree.ipynb

@@ -208,7 +208,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Here, we import a dummy discretisation from the PyBaMM tests directory.\n",
+    "import sys\n",
+    "sys.path.insert(0, pybamm.root_dir())\n",
     "from tests import get_discretisation_for_testing\n",
+    "\n",
     "disc = get_discretisation_for_testing()\n",
     "disc.y_slices = {c: [slice(0, 40)]}\n",
     "dcdt = disc.process_symbol(dcdt)\n",

docs/source/user_guide/installation/index.rst

@@ -152,6 +152,10 @@ Dependency
 ================================================================================ ================== ================== =============================================================
 `pre-commit <https://pre-commit.com/index.html>`__                               \-                 dev                For managing and maintaining multi-language pre-commit hooks.
 `ruff <https://beta.ruff.rs/docs/>`__                                            \-                 dev                For code formatting.
+`nox <https://nox.thea.codes/en/stable/>`__                                      \-                 dev                For running testing sessions in multiple environments.
+`pytest <https://docs.pytest.org/en/stable/>`__                                  6.0.0              dev                For running Jupyter notebooks tests.
+`pytest-xdist <https://pytest-xdist.readthedocs.io/en/latest/>`__                \-                 dev                For running tests in parallel across distributed workers.
+`nbmake <https://github.com/treebeardtech/nbmake/>`__                            \-                 dev                A ``pytest`` plugin for executing Jupyter notebooks.
 ================================================================================ ================== ================== =============================================================
 
 .. _install.cite_dependencies:

docs/source/user_guide/installation/install-from-source.rst

@@ -240,6 +240,7 @@ Doctests, examples, and coverage
 ``Nox`` can also be used to run doctests, run examples, and generate a coverage report using:
 
 - ``nox -s examples``: Run the Jupyter notebooks in ``docs/source/examples/notebooks/``.
+- ``nox -s examples -- <path-to-notebook-1.ipynb> <path-to_notebook-2.ipynb>``: Run specific Jupyter notebooks.
 - ``nox -s scripts``: Run the example scripts in ``examples/scripts/``.
 - ``nox -s doctests``: Run doctests.
 - ``nox -s coverage``: Measure current test coverage and generate a coverage report.
@@ -268,8 +269,8 @@ i.e. ``pip install -e .``. This sets the installed location of the
 source files to your current directory.
 
 **Problem:** Errors when solving model
-``ValueError: Integrator name ida does not exsist``, or
-``ValueError: Integrator name cvode does not exsist``.
+``ValueError: Integrator name ida does not exist``, or
+``ValueError: Integrator name cvode does not exist``.
 
 **Solution:** This could mean that you have not installed
 ``scikits.odes`` correctly, check the instructions given above and make

noxfile.py

@@ -106,8 +106,9 @@ def run_unit(session):
 def run_examples(session):
     """Run the examples tests for Jupyter notebooks."""
     set_environment_variables(PYBAMM_ENV, session=session)
-    session.run_always("pip", "install", "-e", ".[all]")
-    session.run("python", "run-tests.py", "--examples")
+    notebooks_to_test = session.posargs if session.posargs else []
+    session.run_always("pip", "install", "-e", ".[all,dev]")
+    session.run("pytest", "--nbmake", *notebooks_to_test, external=True)
 
 
 @nox.session(name="scripts")

pytest.ini

@@ -0,0 +1,15 @@
+; NOTE: currently used only for notebook tests with the nbmake plugin.
+[pytest]
+; Use pytest-xdist to run tests in parallel by default, exit with
+; error if not installed
+required_plugins = pytest-xdist
+addopts = -nauto -v
+testpaths =
+    docs/source/examples
+console_output_style = progress
+
+; Logging configuration
+log_cli = true
+log_cli_level = INFO
+log_format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+log_date_format = %Y-%m-%d %H:%M:%S