Skip to content
This repository has been archived by the owner on Feb 23, 2024. It is now read-only.

Structure validation #110

Merged
merged 42 commits into from
Jan 23, 2024
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
dec80c1
draft validation functions
viktorpm Oct 12, 2023
566fc44
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 12, 2023
8bb2150
run on all atlases, don't crash on assertion error
viktorpm Oct 12, 2023
8aa73e7
fix merging
viktorpm Oct 12, 2023
9423015
fixing atlas path
viktorpm Nov 2, 2023
1fcafa4
Merge pull request #1 from viktorpm/fix-path
alessandrofelder Nov 2, 2023
319e721
Clearer output printing
alessandrofelder Nov 2, 2023
22d6c96
Merge pull request #2 from viktorpm/minor-tweaks-for-validation
viktorpm Nov 2, 2023
9402b66
tidy up validation script, remove weird test_git
alessandrofelder Nov 3, 2023
598a0e1
add dev install, make test structure, initial tests
alessandrofelder Nov 3, 2023
96c8584
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 3, 2023
ffef504
add tests and return for _assert_close()
viktorpm Nov 3, 2023
92e94e6
add test for validate mesh matches annotation
viktorpm Nov 3, 2023
29343dc
fix linting
viktorpm Nov 3, 2023
5995c43
update version for actions
alessandrofelder Nov 7, 2023
cb9cc02
drop py3.8 in tox, run pytest in tox
alessandrofelder Nov 7, 2023
85419d8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 7, 2023
dc4aebc
fix copy-paste error in pytest command
alessandrofelder Nov 7, 2023
a59da92
drop py3.8 from gh action workflow file too
alessandrofelder Nov 7, 2023
f06c82f
Adding docstrings to validation script
viktorpm Nov 8, 2023
1aeb476
wip: draft structure validation function
viktorpm Nov 29, 2023
dace6b3
Making path tests stricter, breaking up long strings, adding diff_tol…
viktorpm Nov 24, 2023
9faf48f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 29, 2023
777d309
restructuring validate_mesh_matches_image_extents function, adding co…
viktorpm Nov 29, 2023
b8377c5
Merge branch 'validation' of github.com:viktorpm/bg-atlasgen into val…
viktorpm Nov 29, 2023
8736701
testing expected files and meshes directory separately
viktorpm Jan 3, 2024
b517507
looping through validation functions and parameters to catch individu…
viktorpm Jan 8, 2024
26b9dcf
removing hard coded path, generalising to all atlases
viktorpm Jan 9, 2024
f7fa093
adding successful_validations list
viktorpm Jan 10, 2024
af84ec3
tidying up duplications
viktorpm Jan 16, 2024
bd1f185
fix recursive bug
alessandrofelder Jan 16, 2024
2ca9914
checkout finished validate_atlases.py from validation branch
viktorpm Jan 17, 2024
83d3ff0
adding validate_mesh_structure_pairs function
viktorpm Jan 18, 2024
bab77a6
Update bg_atlasgen/validate_atlases.py
viktorpm Jan 22, 2024
6a73af9
adding assertion to validate_mesh_structure_pairs function
viktorpm Jan 22, 2024
7d7d601
checking IDs via bg_atlasapi, checking if IDs have mesh files and acc…
viktorpm Jan 22, 2024
c15a21a
Update bg_atlasgen/validate_atlases.py
viktorpm Jan 22, 2024
21ed954
passing atlas_name to validate_mesh_structure_pairs function
viktorpm Jan 22, 2024
d0f81ab
addressing Niko's final comments, cleaning code
viktorpm Jan 22, 2024
26d7c17
merging changes from validation branch
viktorpm Jan 22, 2024
71527c2
Merge branch 'main' into structure-validation
viktorpm Jan 23, 2024
6183091
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions .github/workflows/test_and_deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: neuroinformatics-unit/actions/lint@v1
- uses: neuroinformatics-unit/actions/lint@v2

test:
needs: lint
Expand All @@ -25,10 +25,8 @@ jobs:
python-version: "3.10"
- os: windows-latest
python-version: "3.9"
- os: ubuntu-latest
python-version: "3.8"

steps:
- uses: neuroinformatics-unit/actions/test@v1
- uses: neuroinformatics-unit/actions/test@v2
with:
python-version: ${{ matrix.python-version }}
21 changes: 0 additions & 21 deletions bg_atlasgen/test_git.py

This file was deleted.

205 changes: 205 additions & 0 deletions bg_atlasgen/validate_atlases.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
"""Script to validate atlases"""


import os
from pathlib import Path

Check warning on line 5 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L4-L5

Added lines #L4 - L5 were not covered by tests

import numpy as np
from bg_atlasapi import BrainGlobeAtlas
from bg_atlasapi.config import get_brainglobe_dir
from bg_atlasapi.list_atlases import (

Check warning on line 10 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L7-L10

Added lines #L7 - L10 were not covered by tests
get_all_atlases_lastversions,
get_atlases_lastversions,
)
from bg_atlasapi.update_atlases import update_atlas

Check warning on line 14 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L14

Added line #L14 was not covered by tests


def validate_atlas_files(atlas_path: Path):

Check warning on line 17 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L17

Added line #L17 was not covered by tests
"""Checks if basic files exist in the atlas folder"""

assert atlas_path.is_dir(), f"Atlas path {atlas_path} not found"
expected_files = [

Check warning on line 21 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L20-L21

Added lines #L20 - L21 were not covered by tests
"annotation.tiff",
"reference.tiff",
"metadata.json",
"structures.json",
]
for expected_file_name in expected_files:
expected_path = Path(atlas_path / expected_file_name)
assert (

Check warning on line 29 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L27-L29

Added lines #L27 - L29 were not covered by tests
expected_path.is_file()
), f"Expected file not found at {expected_path}"

meshes_path = atlas_path / "meshes"
assert meshes_path.is_dir(), f"Meshes path {meshes_path} not found"
return True

Check warning on line 35 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L33-L35

Added lines #L33 - L35 were not covered by tests


def _assert_close(mesh_coord, annotation_coord, pixel_size, diff_tolerance=10):

Check warning on line 38 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L38

Added line #L38 was not covered by tests
"""
Helper function to check if the mesh and the annotation coordinate
are closer to each other than an arbitrary tolerance value times the pixel size.
The default tolerance value is 10.
"""
assert abs(mesh_coord - annotation_coord) <= diff_tolerance * pixel_size, (

Check warning on line 44 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L44

Added line #L44 was not covered by tests
f"Mesh coordinate {mesh_coord} and annotation coordinate {annotation_coord}",
f"differ by more than {diff_tolerance} times pixel size {pixel_size}",
)
return True

Check warning on line 48 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L48

Added line #L48 was not covered by tests


def validate_mesh_matches_image_extents(atlas: BrainGlobeAtlas):

Check warning on line 51 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L51

Added line #L51 was not covered by tests
"""Checks if the mesh and the image extents are similar"""

root_mesh = atlas.mesh_from_structure("root")
annotation_image = atlas.annotation
resolution = atlas.resolution

Check warning on line 56 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L54-L56

Added lines #L54 - L56 were not covered by tests

# minimum and maximum values of the annotation image (z, y, x)
z_range, y_range, x_range = np.nonzero(annotation_image)
z_min, z_max = np.min(z_range), np.max(z_range)
y_min, y_max = np.min(y_range), np.max(y_range)
x_min, x_max = np.min(x_range), np.max(x_range)

Check warning on line 62 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L59-L62

Added lines #L59 - L62 were not covered by tests

# minimum and maximum values of the annotation image scaled by the atlas resolution
z_min_scaled, z_max_scaled = z_min * resolution[0], z_max * resolution[0]
y_min_scaled, y_max_scaled = y_min * resolution[1], y_max * resolution[1]
x_min_scaled, x_max_scaled = x_min * resolution[2], x_max * resolution[2]

Check warning on line 67 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L65-L67

Added lines #L65 - L67 were not covered by tests

# z, y and x coordinates of the root mesh (extent of the whole object)
mesh_points = root_mesh.points
z_coords, y_coords, x_coords = (

Check warning on line 71 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L70-L71

Added lines #L70 - L71 were not covered by tests
mesh_points[:, 0],
mesh_points[:, 1],
mesh_points[:, 2],
)

# minimum and maximum coordinates of the root mesh
z_min_mesh, z_max_mesh = np.min(z_coords), np.max(z_coords)
y_min_mesh, y_max_mesh = np.min(y_coords), np.max(y_coords)
x_min_mesh, x_max_mesh = np.min(x_coords), np.max(x_coords)

Check warning on line 80 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L78-L80

Added lines #L78 - L80 were not covered by tests

# checking if root mesh and image are on the same scale
_assert_close(z_min_mesh, z_min_scaled, resolution[0])
_assert_close(z_max_mesh, z_max_scaled, resolution[0])
_assert_close(y_min_mesh, y_min_scaled, resolution[1])
_assert_close(y_max_mesh, y_max_scaled, resolution[1])
_assert_close(x_min_mesh, x_min_scaled, resolution[2])
_assert_close(x_max_mesh, x_max_scaled, resolution[2])

Check warning on line 88 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L83-L88

Added lines #L83 - L88 were not covered by tests

return True

Check warning on line 90 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L90

Added line #L90 was not covered by tests


def open_for_visual_check():

Check warning on line 93 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L93

Added line #L93 was not covered by tests
# implement visual checks later
pass

Check warning on line 95 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L95

Added line #L95 was not covered by tests


def validate_checksum():

Check warning on line 98 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L98

Added line #L98 was not covered by tests
# implement later
pass

Check warning on line 100 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L100

Added line #L100 was not covered by tests


def check_additional_references():

Check warning on line 103 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L103

Added line #L103 was not covered by tests
# check additional references are different, but have same dimensions
pass

Check warning on line 105 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L105

Added line #L105 was not covered by tests


def validate_mesh_structure_pairs(atlas_name: str, atlas_path: Path):

Check warning on line 108 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L108

Added line #L108 was not covered by tests
# json_path = Path(atlas_path / "structures.json")
atlas = BrainGlobeAtlas(atlas_name)

Check warning on line 110 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L110

Added line #L110 was not covered by tests

obj_path = Path(atlas_path / "meshes")

Check warning on line 112 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L112

Added line #L112 was not covered by tests

ids_from_bg_atlas_api = list(atlas.structures.keys())
ids_from_mesh_files = [

Check warning on line 115 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L114-L115

Added lines #L114 - L115 were not covered by tests
int(Path(file).stem)
for file in os.listdir(obj_path)
if file.endswith(".obj")
]

in_mesh_not_bg = []
for id in ids_from_mesh_files:
if id not in ids_from_bg_atlas_api:
in_mesh_not_bg.append(id)

Check warning on line 124 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L121-L124

Added lines #L121 - L124 were not covered by tests

in_bg_not_mesh = []
for id in ids_from_bg_atlas_api:
if id not in ids_from_mesh_files:
in_bg_not_mesh.append(id)

Check warning on line 129 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L126-L129

Added lines #L126 - L129 were not covered by tests

if len(in_mesh_not_bg) or len(in_bg_not_mesh):
raise AssertionError(

Check warning on line 132 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L131-L132

Added lines #L131 - L132 were not covered by tests
f"Structures with ID {in_bg_not_mesh} are in the atlas, but don't have a corresponding mesh file; "
f"Structures with IDs {in_mesh_not_bg} have a mesh file, but are not accessible through the atlas."
)


def validate_atlas(atlas_name, version, all_validation_functions):

Check warning on line 138 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L138

Added line #L138 was not covered by tests
"""Validates the latest version of a given atlas"""

print(atlas_name, version)
BrainGlobeAtlas(atlas_name)
updated = get_atlases_lastversions()[atlas_name]["updated"]
if not updated:
update_atlas(atlas_name)

Check warning on line 145 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L141-L145

Added lines #L141 - L145 were not covered by tests

validation_function_parameters = [

Check warning on line 147 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L147

Added line #L147 was not covered by tests
# validate_atlas_files(atlas_path: Path)
(Path(get_brainglobe_dir() / f"{atlas_name}_v{version}"),),
# validate_mesh_matches_image_extents(atlas: BrainGlobeAtlas)
(BrainGlobeAtlas(atlas_name),),
# open_for_visual_check()
(),
# validate_checksum()
(),
# check_additional_references()
(),
# validate_mesh_structure_pairs(atlas_name: str, atlas_path: Path):
(
atlas_name,
Path(get_brainglobe_dir() / f"{atlas_name}_v{version}"),
),
]

# list to store the errors of the failed validations
failed_validations = []
successful_validations = []

Check warning on line 167 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L166-L167

Added lines #L166 - L167 were not covered by tests

for i, validation_function in enumerate(all_validation_functions):
try:
validation_function(*validation_function_parameters[i])
successful_validations.append((atlas_name, validation_function))
except AssertionError as error:
failed_validations.append((atlas_name, validation_function, error))

Check warning on line 174 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L169-L174

Added lines #L169 - L174 were not covered by tests

return successful_validations, failed_validations

Check warning on line 176 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L176

Added line #L176 was not covered by tests


if __name__ == "__main__":

Check warning on line 179 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L179

Added line #L179 was not covered by tests
# list to store the validation functions
all_validation_functions = [

Check warning on line 181 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L181

Added line #L181 was not covered by tests
validate_atlas_files,
validate_mesh_matches_image_extents,
open_for_visual_check,
validate_checksum,
check_additional_references,
viktorpm marked this conversation as resolved.
Show resolved Hide resolved
validate_mesh_structure_pairs,
]

valid_atlases = []
invalid_atlases = []
for atlas_name, version in get_all_atlases_lastversions().items():
successful_validations, failed_validations = validate_atlas(

Check warning on line 193 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L190-L193

Added lines #L190 - L193 were not covered by tests
atlas_name, version, all_validation_functions
)
for item in successful_validations:
valid_atlases.append(item)
for item in failed_validations:
invalid_atlases.append(item)

Check warning on line 199 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L196-L199

Added lines #L196 - L199 were not covered by tests

print("Summary")
print("### Valid atlases ###")
print(valid_atlases)
print("### Invalid atlases ###")
print(invalid_atlases)

Check warning on line 205 in bg_atlasgen/validate_atlases.py

View check run for this annotation

Codecov / codecov/patch

bg_atlasgen/validate_atlases.py#L201-L205

Added lines #L201 - L205 were not covered by tests
19 changes: 13 additions & 6 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,19 @@ allenmouse = [
"allensdk",
]

dev = [
"pytest",
"pytest-cov",
"pytest-mock",
"coverage",
"tox",
"black",
"mypy",
"pre-commit",
"ruff",
"setuptools_scm",
]

[build-system]
requires = [
"setuptools>=45",
Expand All @@ -59,12 +72,6 @@ include-package-data = true
[tool.setuptools.packages.find]
include = ["bg_atlasgen*"]

[tool.pytest.ini_options]
addopts = "--cov=bg_atlasgen"
filterwarnings = [
"error",
]


[tool.black]
target-version = ['py38', 'py39', 'py310', 'py311']
Expand Down
Empty file added tests/__init__.py
Empty file.
54 changes: 54 additions & 0 deletions tests/test_unit/test_validation.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
from pathlib import Path

import numpy as np
import pytest
from bg_atlasapi import BrainGlobeAtlas
from bg_atlasapi.config import get_brainglobe_dir

from bg_atlasgen.validate_atlases import (
_assert_close,
validate_atlas_files,
validate_mesh_matches_image_extents,
)


def test_validate_mesh_matches_image_extents():
atlas = BrainGlobeAtlas("allen_mouse_100um")
assert validate_mesh_matches_image_extents(atlas)


def test_validate_mesh_matches_image_extents_negative(mocker):
atlas = BrainGlobeAtlas("allen_mouse_100um")
flipped_annotation_image = np.transpose(atlas.annotation)
mocker.patch(
"bg_atlasapi.BrainGlobeAtlas.annotation",
new_callable=mocker.PropertyMock,
return_value=flipped_annotation_image,
)
with pytest.raises(
AssertionError, match="differ by more than 10 times pixel size"
):
validate_mesh_matches_image_extents(atlas)


def test_valid_atlas_files():
_ = BrainGlobeAtlas("allen_mouse_100um")
atlas_path = Path(get_brainglobe_dir()) / "allen_mouse_100um_v1.2"
assert validate_atlas_files(atlas_path)


def test_invalid_atlas_path():
atlas_path = Path.home()
with pytest.raises(AssertionError, match="Expected file not found"):
validate_atlas_files(atlas_path)


def test_assert_close():
assert _assert_close(99.5, 8, 10)


def test_assert_close_negative():
with pytest.raises(
AssertionError, match="differ by more than 10 times pixel size"
):
_assert_close(99.5, 30, 2)
5 changes: 2 additions & 3 deletions tox.ini
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
[tox]
envlist = py{38,39,310,311}
envlist = py{39,310,311}

[gh-actions]
python =
3.8: py38
3.9: py39
3.10: py310
3.11: py311
Expand All @@ -12,4 +11,4 @@ python =
extras =
dev
commands =
python -c "import bg_atlasgen"
pytest -v --color=yes --cov=bg_atlasgen --cov-report=xml