diff --git a/.gitignore b/.gitignore index c6103dab75..3e01fcac83 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ *.mat *.csv *.hidden +*.pkl # don't ignore important .txt and .csv files !requirements* diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0084f729dc..577dbd67c6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -120,7 +120,9 @@ This allows people to (1) use PyBaMM without ever importing Matplotlib and (2) c All code requires testing. We use the [unittest](https://docs.python.org/3.3/library/unittest.html) package for our tests. (These tests typically just check that the code runs without error, and so, are more _debugging_ than _testing_ in a strict sense. Nevertheless, they are very useful to have!) -If you have nox installed, to run unit tests, type +We also use [pytest](https://docs.pytest.org/en/latest/) along with the [nbmake](https://github.com/treebeardtech/nbmake) and the [pytest-xdist](https://pypi.org/project/pytest-xdist/) plugins to test the example notebooks. + +If you have `nox` installed, to run unit tests, type ```bash nox -s unit @@ -151,26 +153,48 @@ When you commit anything to PyBaMM, these checks will also be run automatically ### Testing the example notebooks -To test all the example notebooks in the `docs/source/examples/` folder, type +To test all the example notebooks in the `docs/source/examples/` folder with `pytest` and `nbmake`, type ```bash nox -s examples ``` -To edit the structure and how the Jupyter notebooks get rendered in the Sphinx documentation (using `nbsphinx`), install [Pandoc](https://pandoc.org/installing.html) on your system, either using `conda` (through the `conda-forge` channel) +Alternatively, you may use `pytest` directly with the `--nbmake` flag: ```bash -conda install -c conda-forge pandoc +pytest --nbmake ``` -or refer to the [Pandoc installation instructions](https://pandoc.org/installing.html) specific to your platform. +which runs all the notebooks in the `docs/source/examples/notebooks/` folder in parallel by default, using the `pytest-xdist` plugin. + +Sometimes, debugging a notebook can be a hassle. To run a single notebook, pass the path to it to `pytest`: + +```bash +pytest --nbmake docs/source/examples/notebooks/notebook-name.ipynb +``` + +or, alternatively, you can use posargs to pass the path to the notebook to `nox`. For example: + +```bash +nox -s examples -- docs/source/examples/notebooks/notebook-name.ipynb +``` + +You may also test multiple notebooks this way. Passing the path to a folder will run all the notebooks in that folder: -If notebooks fail because of changes to PyBaMM, it can be a bit of a hassle to debug. In these cases, you can create a temporary export of a notebook's Python content using +```bash +nox -s examples -- docs/source/examples/notebooks/models/ +``` + +You may also use an appropriate [glob pattern](https://www.malikbrowne.com/blog/a-beginners-guide-glob-patterns) to run all notebooks matching a particular folder or name pattern. + +To edit the structure and how the Jupyter notebooks get rendered in the Sphinx documentation (using `nbsphinx`), install [Pandoc](https://pandoc.org/installing.html) on your system, either using `conda` (through the `conda-forge` channel) ```bash -python run-tests.py --debook docs/source/examples/notebooks/notebook-name.ipynb script.py +conda install -c conda-forge pandoc ``` +or refer to the [Pandoc installation instructions](https://pandoc.org/installing.html) specific to your platform. + ### Testing the example scripts To test all the example scripts in the `examples/` folder, type @@ -242,6 +266,7 @@ This also means that, if you can't fix the bug yourself, it will be much easier ``` This will start the debugger at the point where the `ValueError` was raised, and allow you to investigate further. Sometimes, it is more informative to put the try-except block further up the call stack than exactly where the error is raised. + 2. Warnings. If functions are raising warnings instead of errors, it can be hard to pinpoint where this is coming from. Here, you can use the `warnings` module to convert warnings to errors: ```python @@ -251,6 +276,7 @@ This also means that, if you can't fix the bug yourself, it will be much easier ``` Then you can use a try-except block, as in a., but with, for example, `RuntimeWarning` instead of `ValueError`. + 3. Stepping through the expression tree. Most calls in PyBaMM are operations on [expression trees](https://github.com/pybamm-team/PyBaMM/blob/develop/docs/source/examples/notebooks/expression_tree/expression-tree.ipynb). To view an expression tree in ipython, you can use the `render` command: ```python @@ -258,8 +284,11 @@ This also means that, if you can't fix the bug yourself, it will be much easier ``` You can then step through the expression tree, using the `children` attribute, to pinpoint exactly where a bug is coming from. For example, if `expression_tree.jac(y)` is failing, you can check `expression_tree.children[0].jac(y)`, then `expression_tree.children[0].children[0].jac(y)`, etc. + 3. To isolate whether a bug is in a model, its Jacobian or its simplified version, you can set the `use_jacobian` and/or `use_simplify` attributes of the model to `False` (they are both `True` by default for most models). + 4. If a model isn't giving the answer you expect, you can try comparing it to other models. For example, you can investigate parameter limits in which two models should give the same answer by setting some parameters to be small or zero. The `StandardOutputComparison` class can be used to compare some standard outputs from battery models. + 5. To get more information about what is going on under the hood, and hence understand what is causing the bug, you can set the [logging](https://realpython.com/python-logging/) level to `DEBUG` by adding the following line to your test or script: ```python3 @@ -330,7 +359,7 @@ And then visit the webpage served at http://127.0.0.1:8000. Each time a change t Major PyBaMM features are showcased in [Jupyter notebooks](https://jupyter.org/) stored in the [docs/source/examples directory](docs/source/examples/notebooks). Which features are "major" is of course wholly subjective, so please discuss on GitHub first! -All example notebooks should be listed in [docs/sourceexamples/index.rst](https://github.com/pybamm-team/PyBaMM/blob/develop/docs/source/examples/index.rst). Please follow the (naming and writing) style of existing notebooks where possible. +All example notebooks should be listed in [docs/source/examples/index.rst](https://github.com/pybamm-team/PyBaMM/blob/develop/docs/source/examples/index.rst). Please follow the (naming and writing) style of existing notebooks where possible. All the notebooks are tested daily. diff --git a/docs/source/examples/notebooks/expression_tree/expression-tree.ipynb b/docs/source/examples/notebooks/expression_tree/expression-tree.ipynb index 862357eef5..c860198501 100644 --- a/docs/source/examples/notebooks/expression_tree/expression-tree.ipynb +++ b/docs/source/examples/notebooks/expression_tree/expression-tree.ipynb @@ -208,7 +208,11 @@ "metadata": {}, "outputs": [], "source": [ + "# Here, we import a dummy discretisation from the PyBaMM tests directory.\n", + "import sys\n", + "sys.path.insert(0, pybamm.root_dir())\n", "from tests import get_discretisation_for_testing\n", + "\n", "disc = get_discretisation_for_testing()\n", "disc.y_slices = {c: [slice(0, 40)]}\n", "dcdt = disc.process_symbol(dcdt)\n", diff --git a/docs/source/examples/notebooks/expression_tree/expression_tree1.png b/docs/source/examples/notebooks/expression_tree/expression_tree1.png index c5fdb10553..29d3996358 100644 Binary files a/docs/source/examples/notebooks/expression_tree/expression_tree1.png and b/docs/source/examples/notebooks/expression_tree/expression_tree1.png differ diff --git a/docs/source/examples/notebooks/expression_tree/expression_tree2.png b/docs/source/examples/notebooks/expression_tree/expression_tree2.png index efe66f6570..4c1f219cb9 100644 Binary files a/docs/source/examples/notebooks/expression_tree/expression_tree2.png and b/docs/source/examples/notebooks/expression_tree/expression_tree2.png differ diff --git a/docs/source/examples/notebooks/expression_tree/expression_tree3.png b/docs/source/examples/notebooks/expression_tree/expression_tree3.png index 3f0e022c8b..ad61c7accf 100644 Binary files a/docs/source/examples/notebooks/expression_tree/expression_tree3.png and b/docs/source/examples/notebooks/expression_tree/expression_tree3.png differ diff --git a/docs/source/examples/notebooks/expression_tree/expression_tree4.png b/docs/source/examples/notebooks/expression_tree/expression_tree4.png index 3f0e022c8b..ad61c7accf 100644 Binary files a/docs/source/examples/notebooks/expression_tree/expression_tree4.png and b/docs/source/examples/notebooks/expression_tree/expression_tree4.png differ diff --git a/docs/source/examples/notebooks/expression_tree/expression_tree5.png b/docs/source/examples/notebooks/expression_tree/expression_tree5.png index 6d786c0675..aede10080f 100644 Binary files a/docs/source/examples/notebooks/expression_tree/expression_tree5.png and b/docs/source/examples/notebooks/expression_tree/expression_tree5.png differ diff --git a/docs/source/user_guide/installation/index.rst b/docs/source/user_guide/installation/index.rst index c5714d5d5d..9710a3593a 100644 --- a/docs/source/user_guide/installation/index.rst +++ b/docs/source/user_guide/installation/index.rst @@ -152,6 +152,10 @@ Dependency ================================================================================ ================== ================== ============================================================= `pre-commit `__ \- dev For managing and maintaining multi-language pre-commit hooks. `ruff `__ \- dev For code formatting. +`nox `__ \- dev For running testing sessions in multiple environments. +`pytest `__ 6.0.0 dev For running Jupyter notebooks tests. +`pytest-xdist `__ \- dev For running tests in parallel across distributed workers. +`nbmake `__ \- dev A ``pytest`` plugin for executing Jupyter notebooks. ================================================================================ ================== ================== ============================================================= .. _install.cite_dependencies: diff --git a/docs/source/user_guide/installation/install-from-source.rst b/docs/source/user_guide/installation/install-from-source.rst index ce5d7e0ca3..f8aa4968d9 100644 --- a/docs/source/user_guide/installation/install-from-source.rst +++ b/docs/source/user_guide/installation/install-from-source.rst @@ -240,6 +240,7 @@ Doctests, examples, and coverage ``Nox`` can also be used to run doctests, run examples, and generate a coverage report using: - ``nox -s examples``: Run the Jupyter notebooks in ``docs/source/examples/notebooks/``. +- ``nox -s examples -- ``: Run specific Jupyter notebooks. - ``nox -s scripts``: Run the example scripts in ``examples/scripts/``. - ``nox -s doctests``: Run doctests. - ``nox -s coverage``: Measure current test coverage and generate a coverage report. @@ -268,8 +269,8 @@ i.e. ``pip install -e .``. This sets the installed location of the source files to your current directory. **Problem:** Errors when solving model -``ValueError: Integrator name ida does not exsist``, or -``ValueError: Integrator name cvode does not exsist``. +``ValueError: Integrator name ida does not exist``, or +``ValueError: Integrator name cvode does not exist``. **Solution:** This could mean that you have not installed ``scikits.odes`` correctly, check the instructions given above and make diff --git a/noxfile.py b/noxfile.py index 039742f746..96a0e82809 100644 --- a/noxfile.py +++ b/noxfile.py @@ -106,8 +106,9 @@ def run_unit(session): def run_examples(session): """Run the examples tests for Jupyter notebooks.""" set_environment_variables(PYBAMM_ENV, session=session) - session.run_always("pip", "install", "-e", ".[all]") - session.run("python", "run-tests.py", "--examples") + notebooks_to_test = session.posargs if session.posargs else [] + session.run_always("pip", "install", "-e", ".[all,dev]") + session.run("pytest", "--nbmake", *notebooks_to_test, external=True) @nox.session(name="scripts") diff --git a/pytest.ini b/pytest.ini new file mode 100644 index 0000000000..ac90f5d695 --- /dev/null +++ b/pytest.ini @@ -0,0 +1,15 @@ +; NOTE: currently used only for notebook tests with the nbmake plugin. +[pytest] +; Use pytest-xdist to run tests in parallel by default, exit with +; error if not installed +required_plugins = pytest-xdist +addopts = -nauto -v +testpaths = + docs/source/examples +console_output_style = progress + +; Logging configuration +log_cli = true +log_cli_level = INFO +log_format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" +log_date_format = %Y-%m-%d %H:%M:%S diff --git a/run-tests.py b/run-tests.py index 7adefda3e6..b9d421daa2 100755 --- a/run-tests.py +++ b/run-tests.py @@ -5,7 +5,6 @@ # The code in this file is adapted from Pints # (see https://github.com/pints-team/pints) # -import re import os import shutil import pybamm @@ -97,21 +96,6 @@ def run_doc_tests(): shutil.rmtree("docs/build") -def run_notebooks(executable="python"): - """ - Runs Jupyter notebook tests. Exits if they fail. - """ - - # Scan and run - print("Testing notebooks with executable `" + str(executable) + "`") - - # Test notebooks in docs/source/examples - if not scan_for_notebooks("docs/source/examples", True, executable): - print("\nErrors encountered in notebooks") - sys.exit(1) - print("\nOK") - - def run_scripts(executable="python"): """ Run example scripts tests. Exits if they fail. @@ -128,35 +112,6 @@ def run_scripts(executable="python"): print("\nOK") -def scan_for_notebooks(root, recursive=True, executable="python"): - """ - Scans for, and tests, all notebooks in a directory. - """ - ok = True - debug = False - - # Scan path - for filename in os.listdir(root): - path = os.path.join(root, filename) - - # Recurse into subdirectories - if recursive and os.path.isdir(path): - # Ignore hidden directories - if filename[:1] == ".": - continue - ok &= scan_for_notebooks(path, recursive, executable) - - # Test notebooks - if os.path.splitext(path)[1] == ".ipynb": - if debug: - print(path) - else: - ok &= test_notebook(path, executable) - - # Return True if every notebook is ok - return ok - - def scan_for_scripts(root, recursive=True, executable="python"): """ Scans for, and tests, all scripts in a directory. @@ -186,101 +141,6 @@ def scan_for_scripts(root, recursive=True, executable="python"): return ok -def test_notebook(path, executable="python"): - """ - Tests a single notebook, exits if it doesn't finish. - """ - import nbconvert - import pybamm - - b = pybamm.Timer() - print("Test " + path + " ... ", end="") - sys.stdout.flush() - - # Make sure the notebook has a - # "%pip install pybamm[plot,cite] -q" command, for using Google Colab - # specify UTF-8 encoding otherwise Windows chooses CP1252 by default - # attributed to https://stackoverflow.com/a/49562606 - with open(path, "r", encoding="UTF-8") as f: - if "%pip install pybamm[plot,cite] -q" not in f.read(): - # print error and exit - print("\n" + "-" * 70) - print("ERROR") - print("-" * 70) - print( - "Installation command '%pip install pybamm[plot,cite] -q'" - " not found in notebook" - ) - print("-" * 70) - return False - - # Make sure the notebook has "pybamm.print_citations()" to print the relevant papers - # specify UTF-8 encoding otherwise Windows chooses CP1252 by default - # attributed to https://stackoverflow.com/a/49562606 - with open(path, "r", encoding="UTF-8") as f: - if "pybamm.print_citations()" not in f.read(): - # print error and exit - print("\n" + "-" * 70) - print("ERROR") - print("-" * 70) - print( - "Print citations command 'pybamm.print_citations()' not found in " - "notebook" - ) - print("-" * 70) - return False - - # Load notebook, convert to Python - e = nbconvert.exporters.PythonExporter() - code, __ = e.from_filename(path) - - # Remove coding statement, if present - code = "\n".join([x for x in code.splitlines() if x[:9] != "# coding"]) - - # Tell matplotlib not to produce any figures - env = dict(os.environ) - env["MPLBACKEND"] = "Template" - - # If notebook makes use of magic commands then - # the script must be run using ipython - # https://github.com/jupyter/nbconvert/issues/503#issuecomment-269527834 - executable = ( - "ipython" - if (("run_cell_magic(" in code) or ("run_line_magic(" in code)) - else executable - ) - - # Run in subprocess - cmd = [executable] + ["-c", code] - try: - p = subprocess.Popen( - cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env - ) - stdout, stderr = p.communicate() - # TODO: Use p.communicate(timeout=3600) if Python3 only - if p.returncode != 0: - # Show failing code, output and errors before returning - print("ERROR") - print("-- script " + "-" * (79 - 10)) - for i, line in enumerate(code.splitlines()): - j = str(1 + i) - print(j + " " * (5 - len(j)) + line) - print("-- stdout " + "-" * (79 - 10)) - print(str(stdout, "utf-8")) - print("-- stderr " + "-" * (79 - 10)) - print(str(stderr, "utf-8")) - print("-" * 79) - return False - except KeyboardInterrupt: - p.terminate() - print("ABORTED") - sys.exit(1) - - # Sucessfully run - print("ok ({})".format(b.time())) - return True - - def test_script(path, executable="python"): """ Tests a single script, exits if it doesn't finish. @@ -322,32 +182,6 @@ def test_script(path, executable="python"): return True -def export_notebook(ipath, opath): - """ - Exports the notebook at `ipath` to a Python file at `opath`. - """ - import nbconvert - from traitlets.config import Config - - # Create nbconvert configuration to ignore text cells - c = Config() - c.TemplateExporter.exclude_markdown = True - - # Load notebook, convert to Python - e = nbconvert.exporters.PythonExporter(config=c) - code, __ = e.from_filename(ipath) - - # Remove "In [1]:" comments - r = re.compile(r"(\s*)# In\[([^]]*)\]:(\s)*") - code = r.sub("\n\n", code) - - # Store as executable script file - with open(opath, "w") as f: - f.write("#!/usr/bin/env python") - f.write(code) - os.chmod(opath, 0o775) - - if __name__ == "__main__": # Set up argument parsing parser = argparse.ArgumentParser( @@ -380,10 +214,10 @@ def export_notebook(ipath, opath): parser.add_argument( "--examples", action="store_true", - help="Test all Jupyter notebooks in `docs/source/examples/`.", + help="Test all Jupyter notebooks in `docs/source/examples/` (deprecated, use nox or pytest instead).", # noqa: E501 ) parser.add_argument( - "-debook", + "--debook", nargs=2, metavar=("in", "out"), help="Export a Jupyter notebook to a Python file for manual testing.", @@ -393,7 +227,6 @@ def export_notebook(ipath, opath): "--scripts", action="store_true", help="Test all example scripts in `examples/`.", - ) # Doctests parser.add_argument( @@ -440,13 +273,15 @@ def export_notebook(ipath, opath): if args.doctest: has_run = True run_doc_tests() - # Notebook tests + # Notebook tests (deprecated) elif args.examples: - has_run = True - run_notebooks(interpreter) + raise ValueError( + "Notebook tests are deprecated, use nox -s examples or pytest instead" + ) if args.debook: - has_run = True - export_notebook(*args.debook) + raise ValueError( + "Notebook tests are deprecated, use nox -s examples or pytest instead" + ) # Scripts tests elif args.scripts: has_run = True diff --git a/setup.py b/setup.py index df55a24325..9dd4d41cd3 100644 --- a/setup.py +++ b/setup.py @@ -251,9 +251,16 @@ def compile_KLU(): "tqdm", ], "dev": [ - "pre-commit", # For code style checking - "ruff", # For code style auto-formatting - "nox", # For running testing sessions + # For working with pre-commit hooks + "pre-commit", + # For code style checks: linting and auto-formatting + "ruff", + # For running testing sessions + "nox", + # For testing Jupyter notebooks + "pytest>=6", + "pytest-xdist", + "nbmake", ], "pandas": [ "pandas>=0.24",