[Bug]: 23.9rc0 "optional dependencies" imported

[aabills]

PyBaMM Version

23.9rc0

Python Version

3.11

Describe the bug

There are a number of "optional dependencies" that are imported with pybamm on 23.9rc0, causing errors since they are not installed by default. This can obviously be solved by installing pybamm[all] but it's a bit weird to me that pip install pybamm, then import pybamm doesn't just work.

Not sure if this is a "real bug" or if it will be resolved when 23.9 is actually released. Feel free to close it if it will be resolved.

Steps to Reproduce

shell: pip install pybamm==23.9rc0
python: import pybamm

Relevant log output

No response

[agriyakhetarpal]

Hi @abillscmu, could you try installing v23.9rc0 within a fresh virtual environment and post an error log? I could not reproduce the issue; installing in a virtual environment without any optional dependencies and then importing PyBaMM works fine for me.

[aabills]

❯ conda create -n pybamm-test python=3.11
Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 23.5.2
  latest version: 23.9.0

Please update conda by running

    $ conda update -n base -c defaults conda

Or to minimize the number of packages updated during conda update use

     conda install conda=23.9.0



## Package Plan ##

  environment location: /Users/abills/miniconda3/envs/pybamm-test

  added / updated specs:
    - python=3.11


The following NEW packages will be INSTALLED:

  bzip2              pkgs/main/osx-arm64::bzip2-1.0.8-h620ffc9_4
  ca-certificates    pkgs/main/osx-arm64::ca-certificates-2023.08.22-hca03da5_0
  libffi             pkgs/main/osx-arm64::libffi-3.4.4-hca03da5_0
  ncurses            pkgs/main/osx-arm64::ncurses-6.4-h313beb8_0
  openssl            pkgs/main/osx-arm64::openssl-3.0.11-h1a28f6b_2
  pip                pkgs/main/osx-arm64::pip-23.3-py311hca03da5_0
  python             pkgs/main/osx-arm64::python-3.11.5-hb885b13_0
  readline           pkgs/main/osx-arm64::readline-8.2-h1a28f6b_0
  setuptools         pkgs/main/osx-arm64::setuptools-68.0.0-py311hca03da5_0
  sqlite             pkgs/main/osx-arm64::sqlite-3.41.2-h80987f9_0
  tk                 pkgs/main/osx-arm64::tk-8.6.12-hb8d0fd4_0
  tzdata             pkgs/main/noarch::tzdata-2023c-h04d1e81_0
  wheel              pkgs/main/osx-arm64::wheel-0.41.2-py311hca03da5_0
  xz                 pkgs/main/osx-arm64::xz-5.4.2-h80987f9_0
  zlib               pkgs/main/osx-arm64::zlib-1.2.13-h5a0b063_0


Proceed ([y]/n)? y


Downloading and Extracting Packages

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate pybamm-test
#
# To deactivate an active environment, use
#
#     $ conda deactivate


~ via ஃ v1.9.3 
❯ conda activate pybamm-test

~ via ஃ v1.9.3 via 🅒 pybamm-test 
❯ pip install pybamm==23.9
ERROR: Ignored the following versions that require a different python version: 0.3.0 Requires-Python >=3.6,<3.9; 0.3.0.post1 Requires-Python >=3.6,<3.9; 0.3.0b0 Requires-Python >=3.6,<3.9; 0.4.0 Requires-Python >=3.6,<3.10; 21.10 Requires-Python >=3.7,<3.10; 21.11 Requires-Python >=3.7,<3.10; 21.12 Requires-Python >=3.7,<3.10; 21.8.post1 Requires-Python >=3.6,<3.10; 21.8.post2 Requires-Python >=3.6,<3.10; 21.9 Requires-Python >=3.7,<3.10; 22.1 Requires-Python >=3.7,<3.10; 22.10 Requires-Python >=3.8,<3.10; 22.10.post1 Requires-Python >=3.8,<3.10; 22.11 Requires-Python >=3.8,<3.10; 22.11.1 Requires-Python >=3.8,<3.10; 22.12 Requires-Python >=3.8,<3.10; 22.2 Requires-Python >=3.7,<3.10; 22.3 Requires-Python >=3.7,<3.10; 22.4 Requires-Python >=3.7,<3.10; 22.5 Requires-Python >=3.7,<3.10; 22.6 Requires-Python >=3.7,<3.10; 22.7 Requires-Python >=3.7,<3.10; 22.8 Requires-Python >=3.7,<3.10; 22.9 Requires-Python >=3.7,<3.10; 23.1 Requires-Python >=3.8,<3.10; 23.2 Requires-Python >=3.8,<3.10; 23.3 Requires-Python >=3.8,<3.10
ERROR: Could not find a version that satisfies the requirement pybamm==23.9 (from versions: 0.1.0, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post1, 0.2.2.post2, 0.2.2.post3, 0.2.3, 0.2.3.post1, 0.2.4, 0.2.4.post1, 0.2.4.post2, 0.2.4.post3, 23.4, 23.4.1, 23.5rc0, 23.5, 23.9rc0)
ERROR: No matching distribution found for pybamm==23.9

~ via ஃ v1.9.3 via 🅒 pybamm-test took 2s
❯ pip install pybamm==23.9rc0
Collecting pybamm==23.9rc0
  Using cached pybamm-23.9rc0-py3-none-any.whl
Collecting numpy>=1.16 (from pybamm==23.9rc0)
  Using cached numpy-1.26.1-cp311-cp311-macosx_11_0_arm64.whl.metadata (115 kB)
Collecting scipy>=1.3 (from pybamm==23.9rc0)
  Using cached scipy-1.11.3-cp311-cp311-macosx_12_0_arm64.whl.metadata (165 kB)
Collecting casadi>=3.6.0 (from pybamm==23.9rc0)
  Using cached casadi-3.6.3-cp311-none-macosx_11_0_arm64.whl.metadata (1.8 kB)
Collecting xarray (from pybamm==23.9rc0)
  Using cached xarray-2023.10.1-py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: packaging>=21.3 in ./.local/lib/python3.11/site-packages (from xarray->pybamm==23.9rc0) (23.2)
Collecting pandas>=1.4 (from xarray->pybamm==23.9rc0)
  Using cached pandas-2.1.1-cp311-cp311-macosx_11_0_arm64.whl.metadata (18 kB)
Collecting python-dateutil>=2.8.2 (from pandas>=1.4->xarray->pybamm==23.9rc0)
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2020.1 (from pandas>=1.4->xarray->pybamm==23.9rc0)
  Using cached pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.1 (from pandas>=1.4->xarray->pybamm==23.9rc0)
  Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)
Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas>=1.4->xarray->pybamm==23.9rc0)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Using cached casadi-3.6.3-cp311-none-macosx_11_0_arm64.whl (36.5 MB)
Using cached numpy-1.26.1-cp311-cp311-macosx_11_0_arm64.whl (14.0 MB)
Using cached scipy-1.11.3-cp311-cp311-macosx_12_0_arm64.whl (29.7 MB)
Using cached xarray-2023.10.1-py3-none-any.whl (1.1 MB)
Using cached pandas-2.1.1-cp311-cp311-macosx_11_0_arm64.whl (10.8 MB)
Using cached pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)
Installing collected packages: pytz, tzdata, six, numpy, scipy, python-dateutil, casadi, pandas, xarray, pybamm
Successfully installed casadi-3.6.3 numpy-1.26.1 pandas-2.1.1 pybamm-23.9rc0 python-dateutil-2.8.2 pytz-2023.3.post1 scipy-1.11.3 six-1.16.0 tzdata-2023.3 xarray-2023.10.1

~ via ஃ v1.9.3 via 🅒 pybamm-test took 7s
❯ python
Python 3.11.5 (main, Sep 11 2023, 08:31:25) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pybamm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/abills/miniconda3/envs/pybamm-test/lib/python3.11/site-packages/pybamm/__init__.py", line 55, in <module>
    from .citations import Citations, citations, print_citations
  File "/Users/abills/miniconda3/envs/pybamm-test/lib/python3.11/site-packages/pybamm/citations.py", line 9, in <module>
    import pybtex
ModuleNotFoundError: No module named 'pybtex'
>>>

[agriyakhetarpal]

Thanks, I don't use conda enough to know more about its mechanisms so tagging @arjxn-py here if he can look into it. @abillscmu, could you also try using a virtualenv-based virtual environment and try installing and importing PyBaMM inside it? I don't think there should be a difference but it would still be helpful

[aabills]

I tried it, same thing

[agriyakhetarpal]

To fix: some recommended ways of handling optional dependencies are via 1. importlib plus lazy-loading (finding module once, importing it later) with relevant warnings if it is not installed (https://stackoverflow.com/q/563022/14001839, https://stackoverflow.com/a/52826085/14001839), 2. or perhaps an optionals module (pybamm.optionals) or otherwise associated functions in pybamm.util that may be used as decorators (https://stackoverflow.com/q/27361427/14001839). This would require some extra effort to ensure that all optional dependencies are covered within classes and no imports break (either no top-level imports in a module, or a custom MissingOptionalWarning in pybamm.exceptions which handles this and recommends installing the optional dependency group).

The best way would be one wherein this can be done inside a single file, i.e., __init__.py

[arjxn-py]

I don't use conda enough to know more about its mechanisms

Looking into it, trying to reproduce it with conda.

[arjxn-py]

I was able to reproduce this locally, thanks so much for reporting this @abillscmu
I shall fix this asap, will open a PR for the same hence self-assigning.

[Saransh-cpp]

@abillscmu could you try executing a minimal example by installing pybamm from the linked PR? Everything should work now.

[aabills]

yep, when installing from #3475 it seems to work, thanks