Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark for cellfinder workflow with CLI arguments #24

Merged
merged 24 commits into from
Oct 16, 2023
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
960b328
update gitignore
sfmig Oct 9, 2023
995a783
add asv benchmarks structure
sfmig Sep 21, 2023
c342dbd
edit asv: add force reinstall to pip command and use package from branch
sfmig Sep 21, 2023
19ffcc4
add cellfinder core dependency
sfmig Sep 21, 2023
962e129
add first draft of workflow benchmarks
sfmig Sep 22, 2023
8f231a1
replace setup_cache for setup classmethod approach. add teardown func…
sfmig Sep 22, 2023
2c503d5
small edits to comments on default values
sfmig Sep 22, 2023
675de5c
set sample time to s
sfmig Sep 22, 2023
5bee73c
edit docstring
sfmig Sep 22, 2023
34c0ef0
dependency injection for passing custom CLI arguments during benchmar…
sfmig Oct 9, 2023
1b9c806
change how sys.argv is passed
sfmig Oct 9, 2023
666df24
define logger in a scope common to all setup steps
sfmig Oct 9, 2023
94c9972
skeleton for benchmarking workflow (WIP)
sfmig Oct 9, 2023
e455546
replace setup_cache for cache (WIP)
sfmig Oct 9, 2023
f1f0c9c
define setup_cache to download the data
sfmig Oct 11, 2023
1dc352f
add dosctrings and comments
sfmig Oct 11, 2023
b7ddd73
make input path string
sfmig Oct 11, 2023
db2ac76
add benchmarks for reading with dask
sfmig Oct 11, 2023
774d8cc
add benchmarks for detecting and saving cells
sfmig Oct 11, 2023
5b46de9
import default input json
sfmig Oct 12, 2023
76352ac
remove example
sfmig Oct 12, 2023
c5b02d7
point to main in asv config
sfmig Oct 13, 2023
682c51a
point to correct branch in asv
sfmig Oct 13, 2023
d83f97b
point to the actual branch in asv (Friday brain)
sfmig Oct 13, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,4 @@ venv/

# written by setuptools_scm
**/_version.py
benchmarks/results/*
5 changes: 5 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,8 @@ recursive-exclude * __pycache__
recursive-exclude * *.py[co]
recursive-exclude docs *
recursive-exclude tests *

include *.json
recursive-include benchmarks *.json
recursive-include benchmarks *.py
recursive-exclude benchmarks/results *
194 changes: 194 additions & 0 deletions asv.conf.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
{
// The version of the config file format. Do not change, unless
// you know what you are doing.
"version": 1,

// The name of the project being benchmarked
"project": "brainglobe_workflows",

// The project's homepage
"project_url": "https://github.com/brainglobe/brainglobe-workflows",

// The URL or local path of the source code repository for the
// project being benchmarked
"repo": ".",

// The Python project's subdirectory in your repo. If missing or
// the empty string, the project is assumed to be located at the root
// of the repository.
// "repo_subdir": "",

// Customizable commands for building the project.
// See asv.conf.json documentation.
// To build the package using pyproject.toml (PEP518), uncomment the following lines
"build_command": [
"python -m pip install build",
"python -m build",
"PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}"
],
// To build the package using setuptools and a setup.py file, uncomment the following lines
// "build_command": [
// "python setup.py build",
// "PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}"
// ],

// Customizable commands for installing and uninstalling the project.
// See asv.conf.json documentation.
"install_command": ["in-dir={env_dir} python -mpip install --force-reinstall {wheel_file}"],
"uninstall_command": ["return-code=any python -mpip uninstall -y {project}"],

// List of branches to benchmark. If not provided, defaults to "master"
// (for git) or "default" (for mercurial).
"branches": ["smg/cellfinder-workflow"], // for git
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this right? (It's a different name to the branch of the PR and I had to change it to run it locally?)

Copy link
Contributor Author

@sfmig sfmig Oct 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes sorry, I think that came about from the rebasing and merging of conflicts 🤔
it should have been smg/cellfinder-workflow-cli-w-tests but now that is merged main should work

actually it won't work with main just yet, since I refactored the workflows in this PR 😓

// "branches": ["default"], // for mercurial

// The DVCS being used. If not set, it will be automatically
// determined from "repo" by looking at the protocol in the URL
// (if remote), or by looking for special directories, such as
// ".git" (if local).
// "dvcs": "git",

// The tool to use to create environments. May be "conda",
// "virtualenv", "mamba" (above 3.8)
// or other value depending on the plugins in use.
// If missing or the empty string, the tool will be automatically
// determined by looking for tools on the PATH environment
// variable.
"environment_type": "conda",

// timeout in seconds for installing any dependencies in environment
// defaults to 10 min
//"install_timeout": 600,

// the base URL to show a commit for the project.
"show_commit_url": "https://github.com/brainglobe/brainglobe-workflows/commit/",

// The Pythons you'd like to test against. If not provided, defaults
// to the current version of Python used to run `asv`.
"pythons": ["3.10"],

// The list of conda channel names to be searched for benchmark
// dependency packages in the specified order
"conda_channels": ["conda-forge", "defaults"],

// A conda environment file that is used for environment creation.
// "conda_environment_file": "environment.yml",

// The matrix of dependencies to test. Each key of the "req"
// requirements dictionary is the name of a package (in PyPI) and
// the values are version numbers. An empty list or empty string
// indicates to just test against the default (latest)
// version. null indicates that the package is to not be
// installed. If the package to be tested is only available from
// PyPi, and the 'environment_type' is conda, then you can preface
// the package name by 'pip+', and the package will be installed
// via pip (with all the conda available packages installed first,
// followed by the pip installed packages).
//
// The ``@env`` and ``@env_nobuild`` keys contain the matrix of
// environment variables to pass to build and benchmark commands.
// An environment will be created for every combination of the
// cartesian product of the "@env" variables in this matrix.
// Variables in "@env_nobuild" will be passed to every environment
// during the benchmark phase, but will not trigger creation of
// new environments. A value of ``null`` means that the variable
// will not be set for the current combination.
//
// "matrix": {
// "req": {
// "numpy": ["1.6", "1.7"],
// "six": ["", null], // test with and without six installed
// "pip+emcee": [""] // emcee is only available for install with pip.
// },
// "env": {"ENV_VAR_1": ["val1", "val2"]},
// "env_nobuild": {"ENV_VAR_2": ["val3", null]},
// },

// Combinations of libraries/python versions can be excluded/included
// from the set to test. Each entry is a dictionary containing additional
// key-value pairs to include/exclude.
//
// An exclude entry excludes entries where all values match. The
// values are regexps that should match the whole string.
//
// An include entry adds an environment. Only the packages listed
// are installed. The 'python' key is required. The exclude rules
// do not apply to includes.
//
// In addition to package names, the following keys are available:
//
// - python
// Python version, as in the *pythons* variable above.
// - environment_type
// Environment type, as above.
// - sys_platform
// Platform, as in sys.platform. Possible values for the common
// cases: 'linux2', 'win32', 'cygwin', 'darwin'.
// - req
// Required packages
// - env
// Environment variables
// - env_nobuild
// Non-build environment variables
//
// "exclude": [
// {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows
// {"environment_type": "conda", "req": {"six": null}}, // don't run without six on conda
// {"env": {"ENV_VAR_1": "val2"}}, // skip val2 for ENV_VAR_1
// ],
//
// "include": [
// // additional env for python2.7
// {"python": "2.7", "req": {"numpy": "1.8"}, "env_nobuild": {"FOO": "123"}},
// // additional env if run on windows+conda
// {"platform": "win32", "environment_type": "conda", "python": "2.7", "req": {"libpython": ""}},
// ],

// The directory (relative to the current directory) that benchmarks are
// stored in. If not provided, defaults to "benchmarks"
// "benchmark_dir": "benchmarks",

// The directory (relative to the current directory) to cache the Python
// environments in. If not provided, defaults to "env"
"env_dir": ".asv/env",

// The directory (relative to the current directory) that raw benchmark
// results are stored in. If not provided, defaults to "results".
"results_dir": "benchmarks/results",

// The directory (relative to the current directory) that the html tree
// should be written to. If not provided, defaults to "html".
"html_dir": "benchmarks/html",

// The number of characters to retain in the commit hashes.
// "hash_length": 8,

// `asv` will cache results of the recent builds in each
// environment, making them faster to install next time. This is
// the number of builds to keep, per environment.
"build_cache_size": 2,

// The commits after which the regression search in `asv publish`
// should start looking for regressions. Dictionary whose keys are
// regexps matching to benchmark names, and values corresponding to
// the commit (exclusive) after which to start looking for
// regressions. The default is to start from the first commit
// with results. If the commit is `null`, regression detection is
// skipped for the matching benchmark.
//
// "regressions_first_commits": {
// "some_benchmark": "352cdf", // Consider regressions only after this commit
// "another_benchmark": null, // Skip regression detection altogether
// },

// The thresholds for relative change in results, after which `asv
// publish` starts reporting regressions. Dictionary of the same
// form as in ``regressions_first_commits``, with values
// indicating the thresholds. If multiple entries match, the
// maximum is taken. If no entry matches, the default is 5%.
//
// "regressions_thresholds": {
// "some_benchmark": 0.01, // Threshold of 1%
// "another_benchmark": 0.5, // Threshold of 50%
// },
}
Empty file added benchmarks/__init__.py
Empty file.
Loading