Remove test dependency duplication #620

Job	Run time
Check Linting	15s
Check Manifest	19s
Run tests with numba disabled	23m 57s
Run brainmapper tests to check for breakages	14m 38s
Run package tests (ubuntu-latest, 3.9)	6m 19s
Run package tests (ubuntu-latest, 3.10)	7m 22s
Run package tests (ubuntu-latest, 3.11)	6m 49s
Run package tests (macos-13, 3.11)	6m 17s
Run package tests (macos-latest, 3.11)	4m 20s
Run package tests (windows-latest, 3.11)	7m 46s
Build source distribution and wheel	0s
Publish build distributions	0s
	1h 18m 2s

Provide feedback