It/keras3 pytorch (#396)

* replace tensorflow Tensor with keras tensor * add case for TF prep in prep_model_weights * add different backends to pyproject.toml * add backend configuration to cellfinder init file. tests passing with jax locally * define extra dependencies for cellfinder with different backends. run tox with TF backend * run tox using TF and JAX backend * install TF in brainmapper environment before running tests in CI * add backends check to cellfinder init file * clean up comments * fix tf-nightly import check * specify TF backend in include guard check * clarify comment * remove 'backend' from dependencies specifications * Apply suggestions from code review Co-authored-by: Igor Tatarnikov <[email protected]> * PyTorch runs utilizing multiple cores * PyTorch fix with default models * Tests run on every push for now * Run test on torch backend only * Fixed guard test to set torch as KERAS_BACKEND * KERAS_BACKEND env variable set directly in test_include_guard.yaml * Run test on python 3.11 * Remove tf-nightly from __init__ version check * Added 3.11 to legacy tox config * Changed legacy tox config for real this time * Don't set the wrong max_processing value * Torch is now set as the default backend * Tests only run with torch, updated comments * Unpinned torch version * Add codecov token (#403) * add codecov token * generate xml coverage report * add timeout to testing jobs * Allow turning off classification or detection in GUI (#402) * Allow turning off classification or detection in GUI. * Fix test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor to fix code analysis errors. * Ensure array is always 2d. * Apply suggestions from code review Co-authored-by: Igor Tatarnikov <[email protected]> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Igor Tatarnikov <[email protected]> * Support single z-stack tif file for input (#397) * Support single z-stack tif file for input. * Fix commit hook. * Apply review suggestions. * Remove modular asv benchmarks (#406) * remove modular asv benchmarks * recover old structure * remove asv-specific lines from gitignore and manifest * prune benchmarks * Adapt CI so it covers both new and old Macs, and installs required additional dependencies on M1 (#408) * naive attempt at adapting to silicon mac CI * run include guard test on Silicon CI * double-check hdf5 is needed * Optimize cell detection (#398) (#407) * Replace coord map values with numba list/tuple for optim. * Switch to fortran layout for faster update of last dim. * Cache kernel. * jit ball filter. * Put z as first axis to speed z rolling (row-major memory). * Unroll recursion (no perf impact either way). * Parallelize cell cluster splitting. * Parallelize walking for full images. * Cleanup docs and pep8 etc. * Add pre-commit fixes. * Fix parallel always being selected and numba function 1st class warning. * Run hook. * Older python needs Union instead of |. * Accept review suggestion. * Address review changes. * num_threads must be an int. --------- Co-authored-by: Matt Einhorn <[email protected]> * [pre-commit.ci] pre-commit autoupdate (#412) updates: - [github.com/pre-commit/pre-commit-hooks: v4.5.0 → v4.6.0](pre-commit/pre-commit-hooks@v4.5.0...v4.6.0) - [github.com/astral-sh/ruff-pre-commit: v0.3.5 → v0.4.3](astral-sh/ruff-pre-commit@v0.3.5...v0.4.3) - [github.com/psf/black: 24.3.0 → 24.4.2](psf/black@24.3.0...24.4.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: sfmig <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Simplify model download (#414) * Simplify model download * Update model cache * Remove jax and tf tests * Standardise the data types for inputs to all be float32 * Force torch to use CPU on arm based macOS during tests * Added PYTORCH_MPS_HIGH_WATERMARK_RATION env variable * Set env variables in test setup * Try to set the default device to cpu in the test itself * Add device call to Conv3D to force cpu * Revert changes, request one cpu left free * Revers the numb cores, don't use arm based mac runner * Merged main, removed torch flags on cellfinder install for guards and brainmapper * Lowercase Torch * Change cache directory --------- Co-authored-by: sfmig <[email protected]> Co-authored-by: Kimberly Meechan <[email protected]> Co-authored-by: Matt Einhorn <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Alessandro Felder <[email protected]> Co-authored-by: Adam Tyson <[email protected]>
brainglobe · May 10, 2024 · 5a0152b · 5a0152b
1 parent 32a0a56
commit 5a0152b
Show file tree

Hide file tree

Showing 42 changed files with 984 additions and 991 deletions.
diff --git a/.github/workflows/test_and_deploy.yml b/.github/workflows/test_and_deploy.yml
@@ -41,10 +41,10 @@ jobs:
       matrix:
         # Run all supported Python versions on linux
         os: [ubuntu-latest]
-        python-version: ["3.9", "3.10"]
-        # Include one macos run
+        python-version: ["3.9", "3.10", "3.11"]
+        # Include one windows, one macos run (intel based macOS 13 runner)
         include:
-        - os: macos-latest
+        - os: macos-13
           python-version: "3.10"
         - os: windows-latest
           python-version: "3.10"
@@ -54,8 +54,8 @@ jobs:
       - name: Cache Keras model
         uses: actions/cache@v3
         with:
-          path: "~/.cellfinder"
-          key: models-${{ hashFiles('~/.cellfinder/**') }}
+          path: "~/.brainglobe"
+          key: models-${{ hashFiles('~/.brainglobe/**') }}
       # Setup pyqt libraries
       - name: Setup qtpy libraries
         uses: tlambert03/setup-qt-libs@v1
@@ -65,11 +65,13 @@ jobs:
       - uses: neuroinformatics-unit/actions/test@v2
         with:
           python-version: ${{ matrix.python-version }}
+          secret-codecov-token: ${{ secrets.CODECOV_TOKEN }}
           use-xvfb: true
 
   test_numba_disabled:
     needs: [linting, manifest]
     name: Run tests with numba disabled
+    timeout-minutes: 60
     runs-on: ubuntu-latest
     env:
        NUMBA_DISABLE_JIT: "1"
@@ -78,8 +80,8 @@ jobs:
       - name: Cache Keras model
         uses: actions/cache@v3
         with:
-          path: "~/.cellfinder"
-          key: models-${{ hashFiles('~/.cellfinder/**') }}
+          path: "~/.brainglobe"
+          key: models-${{ hashFiles('~/.brainglobe/**') }}
       # Setup pyqt libraries
       - name: Setup qtpy libraries
         uses: tlambert03/setup-qt-libs@v1
@@ -89,22 +91,24 @@ jobs:
       - uses: neuroinformatics-unit/actions/test@v2
         with:
           python-version: "3.10"
+          secret-codecov-token: ${{ secrets.CODECOV_TOKEN }}
           codecov-flags: "numba"
 
   # Run brainglobe-workflows brainmapper-CLI tests to check for
   # breakages
   test_brainmapper_cli:
     needs: [linting, manifest]
     name: Run brainmapper tests to check for breakages
+    timeout-minutes: 60
     runs-on: ubuntu-latest
     env:
-      KERAS_BACKEND: jax
+      KERAS_BACKEND: torch
     steps:
       - name: Cache Keras model
         uses: actions/cache@v3
         with:
-          path: "~/.cellfinder"
-          key: models-${{ hashFiles('~/.cellfinder/**') }}
+          path: "~/.brainglobe"
+          key: models-${{ hashFiles('~/.brainglobe/**') }}
 
       - name: Checkout brainglobe-workflows
         uses: actions/checkout@v3
@@ -119,8 +123,9 @@ jobs:
       - name: Install test dependencies
         run: |
           python -m pip install --upgrade pip wheel
-          # Install cellfinder from the latest SHA on this branch (Keras with JAX backend)
-          python -m pip install "cellfinder[jax] @ git+$GITHUB_SERVER_URL/$GITHUB_REPOSITORY@$GITHUB_SHA"
+          # Install cellfinder from the latest SHA on this branch
+          python -m pip install "cellfinder @ git+$GITHUB_SERVER_URL/$GITHUB_REPOSITORY@$GITHUB_SHA"
+
           # Install checked out copy of brainglobe-workflows
           python -m pip install .[dev]
 

diff --git a/.github/workflows/test_include_guard.yaml b/.github/workflows/test_include_guard.yaml
@@ -24,11 +24,13 @@ jobs:
         with:
           python-version: '3.10'
 
-      - name: Install cellfinder via pip, specifying tensorflow as keras' backend
-        run: python -m pip install -e ".[tf]"
+      - name: Install cellfinder via pip
+        run: python -m pip install -e "."
 
       - name: Test (working) import
         uses: jannekem/run-python-script-action@v1
+        env:
+          KERAS_BACKEND: torch
         with:
           fail-on-error: true
           script: |

diff --git a/.gitignore b/.gitignore
@@ -130,11 +130,6 @@ mprofile*.dat
 
 *.DS_Store
 
-# asv
-.asv
-benchmarks/results
-benchmarks/html
-benchmarks/env
 
 # OS
 .DS_Store

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -4,7 +4,7 @@ ci:
 
 repos:
     - repo: https://github.com/pre-commit/pre-commit-hooks
-      rev: v4.5.0
+      rev: v4.6.0
       hooks:
           - id: check-docstring-first
           - id: check-executables-have-shebangs
@@ -16,10 +16,10 @@ repos:
           - id: requirements-txt-fixer
           - id: trailing-whitespace
     - repo: https://github.com/astral-sh/ruff-pre-commit
-      rev: v0.3.5
+      rev: v0.4.3
       hooks:
         - id: ruff
     - repo: https://github.com/psf/black
-      rev: 24.3.0
+      rev: 24.4.2
       hooks:
           - id: black
diff --git a/MANIFEST.in b/MANIFEST.in
@@ -12,7 +12,7 @@ exclude tox.ini
 graft cellfinder
 include cellfinder/napari/napari.yaml
 
-prune benchmarks
 prune examples
 prune resources
 prune tests
+prune benchmarks
diff --git a/benchmarks/README.md b/benchmarks/README.md
@@ -1,39 +1,12 @@
-# Benchmarking with asv
-[Install asv](https://asv.readthedocs.io/en/stable/installing.html) by running:
-```
-pip install asv
-```
-
-`asv` works roughly as follows:
-1. It creates a virtual environment (as defined in the config)
-2. It installs the software package version of a specific commit (or of a local commit)
-3. It times the benchmarking tests and saves the results to json files
-4. The json files are 'published' into an html dir
-5. The html dir can be visualised in a static website
-
-## Running benchmarks
-To run benchmarks on a specific commit:
-```
-$ asv run 88fbbc33^!
-```
-
-To run them up to a specific commit:
-```
-$ asv run 88fbbc33
-```
-
-To run them on a range of commits:
-```
-$ asv run 827f322b..729abcf3
-```
-
-To collate the benchmarks' results into a viewable website:
-```
-$ asv publish
-```
-This will create a tree of files in the `html` directory, but this cannot be viewed directly from the local filesystem, so we need to put them in a static site. `asv publish` also detects statistically significant decreases of performance, the results can be inspected in the 'Regression' tab of the static site.
-
-To visualise the results in a static site:
-```
-$ asv preview
-```
+# Benchmarks
+`detect_and_classify.py` contains a simple script that runs
+detection and classification with the small test dataset.
+
+## Memory
+[memory_profiler](https://github.com/pythonprofilers/memory_profiler)
+can be used to profile memory useage. Install, and then run
+`mprof run --include-children --multiprocess detect_and_classify.py`. It is **very**
+important to use these two flags to capture memory usage by the additional
+processes that cellfinder.core uses.
+
+To show the results of the latest profile run, run `mprof plot`.
diff --git a/benchmarks/asv.conf.json b/benchmarks/asv.conf.json
diff --git a/benchmarks/benchmarks/__init__.py b/benchmarks/benchmarks/__init__.py