Expand the developer documentation (#144)

* Write high-level intro to codebase

intro contains
- guiding principles (with explanatory examples)
- naming and organisation of packages in org
- suggested default software architecture for brainglobe tools

Also this commit
- incorporates the intro into the wider website
- adds some knowledge that was living in our heads to website text.

* minor tweaks

* ignore sign-in only link

* document axis ordering conventions

* link to core dev project board

* document release process and frequency

* document handling of test data and user folders

* link to GH for tools

* make linkcheck happy

* Apply suggestions from @willGraham01's code review

Co-authored-by: Will Graham <[email protected]>

* Apply suggestions from @viktorpm 's code review

* fix typo

* Apply suggestions from code review

Co-authored-by: Adam Tyson <[email protected]>

* minor doc tweaks following review

* avoid direct section ref

* small reword of user data section

---------

Co-authored-by: Will Graham <[email protected]>
Co-authored-by: Adam Tyson <[email protected]>

docs/source/community/developers/conventions.md

@@ -14,3 +14,7 @@ determine the **minimum** set of supported package versions:
 
 In addition to this, the last 24 months of other dependencies should also be
 supported.
+
+## Axis ordering in spatial arrays
+
+For arrays representing spatial data, we follow `zyx` axis ordering in the same way as `numpy` and `napari`. The origin is the upper left corner when you show the first element `stack[0, :, :]`. The first dimension is the one that you are slicing, the second is the height of the image, and the third is the width of the image. The [`brainglobe-space` package](/documentation/brainglobe-space/index.md) provides an interface to manipulate data following different conventions to adhere to this standard.

docs/source/community/developers/index.md

@@ -6,8 +6,10 @@
 
 There are many BrainGlobe repositories, so it may not be obvious where a new contribution should go.
 If you're unsure about any part of the contributing process, please [get in touch](../../contact.md). 
+
 The best place for questions about contributing is probably
 the [BrainGlobe Zulip chat](https://brainglobe.zulipchat.com/).
+You are furthermore welcome to join the bi-weekly developer meetings and contribute items to the agenda - check out the [developer-meeting stream on Zulip](https://brainglobe.zulipchat.com/#narrow/stream/414089-developer-meeting) (requires sign-up) for more information.
 
 If for any reason, you'd rather not reach out in public, feel free to send a direct message on Zulip 
 to [Adam Tyson](https://github.com/adamltyson), one of the core developers.
@@ -19,10 +21,20 @@ You can view these repositories and the relevant information by heading to the [
 
 To add a new BrainGlobe atlas, please see the guide [here](/documentation/bg-atlasapi/adding-a-new-atlas).
 
-## Creating a development environment
+## To contribute code
+
+Before contributing code, it may be useful to familiarise yourself with the [introduction to the BrainGlobe code for developers](./intro_to_codebase.md) as well as the [testing](./testing.md), [developer tooling](./tooling.md) and [conventions](./conventions.md) sections. 
+
+The core development team will support you in contributing code, irrespective of your experience.
+To ensure BrainGlobe remains easy-to-maintain, they will help ensure all code contributions meet 
+a high standard.
+
+
+### Creating a development environment
 
-It is recommended to use `conda` to install a development environment for
-BrainGlobe projects. Once you have `conda` installed, the following commands
+It is recommended to use a recent version of `conda` to install a development environment for
+BrainGlobe projects ([`conda` versions >=23.10.0](https://conda.org/blog/2023-11-06-conda-23-10-0-release/) 
+will significantly speed up installation time). Once you have `conda` installed, the following commands
 will create and activate a `conda` environment with the requirements needed
 for a development environment:
 
@@ -51,7 +63,7 @@ pip install -e '.[dev]'
 from inside the repository. This will install the package, its dependencies,
 and its development dependencies.
 
-## Pull requests
+### Pull requests
 
 In all cases, please submit code to the main repository via a pull request. The developers recommend, and adhere,
 to the following conventions:
@@ -93,6 +105,7 @@ If you aren't sure where the changes should be made, please
 
 :::{toctree}
 :maxdepth: 1
+intro_to_codebase
 tooling
 conventions
 testing

docs/source/community/developers/intro_to_codebase.md

@@ -0,0 +1,81 @@
+# Introduction to the BrainGlobe codebase for developers
+
+This is an introduction to the high-level organisation of the BrainGlobe codebase for developers.
+It serves as an introduction for new contributors and as a reference for all contributors.
+
+It will cover BrainGlobe's guiding principles for development and the default software architecture for BrainGlobe repositories.
+Note that this high-level organisation is (at least partially) aspirational, and implementing it is work-in-progress (see [the current roadmap](/community/roadmaps/index.md) and the [Core Development Project Board](https://github.com/orgs/brainglobe/projects/2))
+
+
+## Guiding principles for development
+
+The first guiding principle is that BrainGlobe should be easy-to-use by everyone.
+In practice, this means that BrainGlobe code should be independent of species of interest or image modality, and should be installable and runnable by anyone with a reasonably modern laptop within minutes.
+Users should be able to achieve their aims regardless of their level of programming expertise.
+This means we aim to cater to a range of potential users.
+
+As a secondary guiding principle, we additionally aim to make the codebase **easy to maintain**, and **easy to contribute to**, where this doesn't interfere with the first principle.
+
+Choices around the software architecture and technology stack (detailed below) are taken with these principles in mind.
+
+
+### Examples of guiding principles in practice
+
+* Ease of installation: through the metapackage, we provide a one-line command to install all BrainGlobe tools at once. None of the packages depend on anything other than Python (we've removed historical compiled code), and are therefore easy to install cross-platform.
+* Accessibility: we aim to provide a Graphical User Interface (GUI) for all BrainGlobe tools. By asking users for feedback, we ensure that the GUI provides a nice user experience.
+* Ease of use through Python/interoperability: we aim to provide a well-documented Python API for all BrainGlobe tools.
+* Performance: By running weekly benchmarks comparing the latest release with the development version, we guarantee that performance will not deteriorate as BrainGlobe evolves.
+* Species/modality independent: none of the code makes any assumptions about the imaging modality or the species of the model organism of interest. We provide atlases for a variety of model organisms.
+* Useability: we sacrifice the code simplicity provided by `magicgui` in exchange for fine-grained control of the user experience by writing brainglobe widgets in `qtpy`. This is an example where the first guiding principle takes priority over the second.
+* Easy-to-maintain: we move functionality used by more than one independent BrainGlobe tool to `brainglobe-utils` to reduce code duplication and make maintenance easier.
+
+
+## BrainGlobe Tools
+
+Code providing functionality related to a specific analysis or visualisation step is referred to as a BrainGlobe "tool".
+Each BrainGlobe tool has its own Github repository on the BrainGlobe organisation. 
+
+Currently stable tools are:
+- [`brainglobe-atlasapi`](https://github.com/brainglobe/bg-atlasapi)
+- [`brainglobe-heatmap`](https://github.com/brainglobe/brainglobe-heatmap)
+- [`brainglobe-segmentation`](https://github.com/brainglobe/brainglobe-segmentation)
+- [`brainreg`](https://github.com/brainglobe/brainreg)
+- [`brainrender`](https://github.com/brainglobe/brainrender)
+- [`cellfinder`](https://github.com/brainglobe/cellfinder)
+- [`morphapi`](https://github.com/brainglobe/morphapi)
+- [`brainglobe-space`](https://github.com/brainglobe/brainglobe-space)
+
+Tools currently in development are
+- [`brainglobe-registration`](https://github.com/brainglobe/brainglobe-registration)
+- [`brainrender-napari`](https://github.com/brainglobe/brainrender-napari)
+
+The BrainGlobe Github organisation also hosts the [`brainglobe` (meta-)package](./repositories/brainglobe-meta/index.md) and the [`brainglobe-workflows` collection](./repositories/brainglobe-workflows/index.md) in separate repositories (which are not tools in themselves), as well as the [utility package `brainglobe-utils`](https://github.com/brainglobe/brainglobe-utils).
+
+As can be seen from the package names, we follow a **loose** naming convention for packages following a pattern of {brain(globe) || cell}-{noun describing what the tool does}. The most important criterion is the expressiveness of the name.
+The `bg-` prefix for BrainGlobe tools has been discontinued.
+
+### User data
+
+User data is stored locally in hidden folders, usually in the user's `$HOME` directory.
+These folders are named after the tools (e.g. `~/.cellfinder/`) or in appropriate subfolders of `.brainglobe` (e.g. `~/.brainglobe/mpin_zfish_1um_v1.0/`).
+We plan to move all user data to `~/.brainglobe` in the future, and this is therefore the place to add new kinds of user data (in appropriately named subfolders).
+Data that we provide (e.g. atlas data and test data) should be hosted on [GIN/BrainGlobe](https://gin.g-node.org/BrainGlobe/), and not on GitHub itself (unless it's a small package-specific text file, in which case it can go in [the package resources and accessed via `importlib`](https://docs.python.org/3/library/importlib.resources.html)).
+We rely on [the `pooch` package](https://www.fatiando.org/pooch/latest/) to fetch data from GIN.
+
+### Default architecture for BrainGlobe Tools
+
+By default, each BrainGlobe tool should be organised into up to three distinct submodules: `core`, `qt`, and `napari`.
+These submodules should live in the same GitHub repository and are packaged together on PyPI.
+It is acceptable to deviate from the default where there is a reason to (e.g. not all tools will have a GUI).
+
+* `core` contains the central logic to use this tool and exposes it through a Python API
+* `qt` contains user-friendly widgets and related Qt code that is implemented using `qtpy`. Each of these widgets provides an intuitive graphical user interface for one part (or few related parts) of the Python API defined in `core`. 
+The widgets additionally emit Qt signals that any frontend (e.g. the code in `napari`) can connect to.
+* `napari` contains at least one Napari plugin that connects to signals in `qt` and implements a napari-specific response (e.g. adding a Napari layer) to them.
+
+The architecture has a number of advantages;
+- The modularity ensure each of submodule can (and should be) tested individually in the first instance
+- Widgets can be re-used outside of napari
+- Integration tests in the same repo to avoid messy CI dependencies
+
+Additionally, keeping the architecture consistent across BrainGlobe tools should make it easier to contribute to several tools once someone has contributed to one tool, due to the familiar codebase organisation.

docs/source/community/developers/new_releases.md

@@ -7,6 +7,20 @@ Maintainers can trigger a new release by pushing a new tag, in the format `vX.Y.
 The `v` prefix **is necessary** as the workflow will only attempt to upload to `PyPI` if the tag matches the format previously provided.
 The `X`, `Y`, and `Z` values should be integers corresponding to the new version number.
 
+## Coordinating releases with the documentation and the metapackage
+
+Releases will be made ad-hoc as bug-fixes and new features become available.
+When releasing a new version of a BrainGlobe repository, we also need to update the website, the metapackage, and any other tools that depend on that repository accordingly.
+This means we will typically create at least three dependent PRs;
+- One in the repository itself (containing the bugfix or new feature we'd like to release)
+- One in the [website repository](https://github.com/brainglobe/brainglobe.github.io) (to update documentation if necessary)
+- One for each repository that depends on the updated tool, to bump the dependency version
+- One in the [metapackage repository](https://github.com/brainglobe/brainglobe-meta), which pins the new versions of all affected tools at once
+
+We should cross-link the latter to the website update, and release all affected packages to PyPI (and conda if appropriate) once they are all merged into `main`.
+Ideally, updates and releases should be made in an order that [follows the dependency tree](<project:repositories/brainglobe-meta/index.md#dependency-tree>) - starting with our lower level tools, than their dependents, then dependents of those dependents, and so on.
+The meta-package itself will always be the last by this convention.
+
 ## Triggering a new release
 
 The steps for triggering a new release are:

docs/source/community/developers/repositories/brainglobe-meta/index.md

@@ -69,6 +69,7 @@ This ensures that:
 - Major or breaking changes to BrainGlobe tools are **not** automatically picked up on update!
 - We can force certain (combinations of) minor/patch updates to tools on users by simply releasing minor/patch updates to `brainglobe`. Changing `cellfinder>=1.1.0,<2` to `cellfinder>=1.1.4,<2` and `brainrender>=2.1.3,<3"` to `brainrender>=2.2.1,<3"`, then creating a new `brainglobe` minor/patch release, allows the user to update both of these packages with an update to `brainglobe`, not having to worry about the individual packages themselves.
 
+(dependency_tree)=
 ## Dependency Tree
 
 BrainGlobe comprises a number of tools, spread across a number of repositories.

docs/source/community/developers/testing.md

@@ -27,3 +27,10 @@ from PyPI to be set in repository secrets.
 Many of these workflows use actions from 
 [neuroinformatics-unit/actions](https://github.com/neuroinformatics-unit/actions). Feel free to raise a PR to that 
 repository!
+
+## Test data
+
+Data used by the tests should be kept on [GIN](https://gin.g-node.org/BrainGlobe/) and fetched using `pooch`.
+Test data should not live on GitHub.
+To avoid local tests running checks on user data interfering with separate user data on the same machine, tests should mock test-user data by mocking `Path.home()` - an example of how to achieve this can [be viewed in `brainrender-napari`](https://github.com/brainglobe/brainrender-napari/blob/014f5c5908065ddaa5d6b05ecdf90493383cfa2f/tests/conftest.py). 
+

docs/source/conf.py

@@ -191,4 +191,7 @@
 notfound_urls_prefix = None
 
 
-linkcheck_ignore = ["https://neuromorpho.org/"]
+linkcheck_ignore = [
+    "https://neuromorpho.org/",
+    "https://brainglobe.zulipchat.com/#narrow/stream/414089-developer-meeting"
+    ]