Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First working implementation of the derotoation package #3

Merged
merged 169 commits into from
Dec 19, 2023
Merged
Changes from 1 commit
Commits
Show all changes
169 commits
Select commit Hold shift + click to select a range
e3be111
Find optimal threshold for frames
lauraporta Jun 20, 2023
2cf3636
Assign correct sign for rotation
lauraporta Jun 20, 2023
4f2a0e3
Get the rotation angle per tick
lauraporta Jun 20, 2023
52bea43
Add draft logic to fit the rotation degree changes to curve [not work…
lauraporta Jun 20, 2023
9708a17
Linting
lauraporta Jun 20, 2023
8c843f6
Add working logic to calculate the rotation degree for each frame
lauraporta Jun 22, 2023
2c42d39
Visualize effects of derotation and correct some errors
lauraporta Jun 22, 2023
0bac20f
Add draft pipeline to find centroid
lauraporta Jun 23, 2023
7b4748e
Add libraries to pyproject.toml
lauraporta Jun 26, 2023
510db18
Clean up
lauraporta Jun 26, 2023
6c0e0ca
Improve plots
lauraporta Jun 26, 2023
5c73d88
Optimization v1
lauraporta Jun 26, 2023
3c63528
Add changes to gitignore
lauraporta Jun 28, 2023
2776f7c
Add functioning algorithm to optimize rotation degrees
lauraporta Jun 28, 2023
0be6a4b
Add improvements in the optimization algorithms for centroid location…
lauraporta Jun 29, 2023
a233b24
Add new plotting of centroids
lauraporta Jul 10, 2023
cd4d14f
Move part of the analysis code in methods in other files - analog pre…
lauraporta Jul 10, 2023
4b8596f
Update pyproject.toml
lauraporta Jul 10, 2023
dd681ce
Refactor plots and other methods
lauraporta Jul 10, 2023
b23c008
dummy napari plotting widget
alessandrofelder Jul 10, 2023
50af5aa
Merge branch 'derotation-01' into napari-matplotlib
lauraporta Jul 10, 2023
6c75b50
make a napari-MPL widget
alessandrofelder Jul 10, 2023
b5a6acb
Add functioning napari plugin
lauraporta Jul 10, 2023
ce21d4a
Add dependencies
lauraporta Jul 11, 2023
2ea7dd1
Use unique configs and make path relative
lauraporta Jul 11, 2023
5ceebd8
Big refactoring
lauraporta Jul 11, 2023
3faeff0
Remove curve fitting
lauraporta Jul 11, 2023
e413509
Clean up gitignore
lauraporta Jul 11, 2023
2242c30
Move code to find best centroid
lauraporta Jul 11, 2023
ef07dd4
Rename method
lauraporta Jul 11, 2023
6ee191c
Use pipeline in old script
lauraporta Jul 11, 2023
de727b5
Add new strategy for derotation
lauraporta Jul 11, 2023
e91716e
Small edits
lauraporta Jul 11, 2023
51a59f6
Visualize labels
lauraporta Jul 13, 2023
0e18811
Use pilot data
lauraporta Jul 13, 2023
040c546
Fix imports and attributes
lauraporta Jul 13, 2023
97bf38c
Add less permissive angle
lauraporta Jul 14, 2023
1a46ec4
Rename files
lauraporta Jul 14, 2023
b2a636c
Load grid data
lauraporta Aug 1, 2023
b7227ee
More workarounds to play with grid data 🥲
lauraporta Aug 1, 2023
021ded8
Renaming functions to make them generic and add WORKING DEROTATION BY…
lauraporta Aug 2, 2023
6e2c106
Further refinement and changes in napari and debugging script
lauraporta Aug 2, 2023
bd9fc9d
Improve the algorithm for end of rotation
lauraporta Aug 3, 2023
aa50d66
Improve naming of conditions
lauraporta Aug 3, 2023
ddd1627
Remove bugs for negative rotations
lauraporta Aug 3, 2023
a2a94f7
Handle rotation ticks missing better
lauraporta Aug 3, 2023
b476547
Add script to visualize how different params of scipy.rotate affect t…
lauraporta Aug 3, 2023
9ddd29e
Handle better initialization of original image at the beginning and a…
lauraporta Aug 4, 2023
78380b1
Choose data
lauraporta Aug 4, 2023
c902611
Improve plotting and image handling in script
lauraporta Aug 4, 2023
20f03b4
Add two images regarding scipy rotate params exploration
lauraporta Aug 7, 2023
f45fd98
Fix error in detection of false rotations
lauraporta Aug 7, 2023
26a2aba
Workaround for suboptimal threshold identification
lauraporta Aug 7, 2023
62ac2b5
Clean up napari script
lauraporta Aug 7, 2023
31b9c65
Temporary adjustament of rotation algorithm
lauraporta Aug 7, 2023
6b5226d
Include other datasets
lauraporta Aug 7, 2023
efb478c
Complete pipeline including masking and saving of tif file
lauraporta Aug 7, 2023
d285ba6
Renaming file
lauraporta Aug 9, 2023
45a5caa
Refactoring, move modules in different folders
lauraporta Aug 9, 2023
106e0b8
More refactoring and deletion of unused methods
lauraporta Aug 9, 2023
a3ce9db
More refactoring
lauraporta Aug 9, 2023
7e54f6b
More refactoring related to data in input, changes in the config files
lauraporta Aug 10, 2023
aa0355d
Add readme
lauraporta Aug 10, 2023
1e96559
Renaming
lauraporta Aug 10, 2023
67c2c49
Change dependencies
lauraporta Aug 10, 2023
abfa65c
Add logging
lauraporta Aug 10, 2023
017a7a7
Fix small import bug and change order of rotate method
lauraporta Aug 10, 2023
0a9ec55
Upadate readme
lauraporta Aug 10, 2023
a8746ee
Fix wrong log name
lauraporta Aug 10, 2023
9cc123a
Edits to the readme
lauraporta Aug 10, 2023
2ce5b0b
Add extra method (draft)
lauraporta Sep 10, 2023
17a9bb7
Add script to saved tifs for unique rotations
lauraporta Sep 10, 2023
f84a43b
Add script to correct an artefact
lauraporta Oct 4, 2023
b650c36
Add script to remove rotations for completed tif file
lauraporta Oct 4, 2023
82f7bf3
Move script to isolate rotations
lauraporta Oct 4, 2023
6e13a10
Add other plotting scripts
lauraporta Oct 10, 2023
7d3e35c
Get data retreives file name
lauraporta Oct 10, 2023
eed8414
Plots are not saved here anymore
lauraporta Oct 10, 2023
90d47b4
Improve rotation by line and add fileter for incremental rotation
lauraporta Oct 10, 2023
608e467
Delete older plots
lauraporta Oct 10, 2023
ca0b046
Load data refactoring
lauraporta Oct 13, 2023
c54869a
Load data refactoring
lauraporta Oct 13, 2023
6028ba4
Merge commit 'ca0b046c3c9df1f65318cacf0256cc96231a053d' into code-cle…
lauraporta Oct 13, 2023
5c63f1c
Remove hard coded values
lauraporta Oct 13, 2023
a0a8507
Add improvements to analog signals processing and a test
lauraporta Oct 17, 2023
2e1c1f5
Add debugging plots, more logging and reformatting of checks
lauraporta Oct 23, 2023
a6c4850
Remove method with bisect
lauraporta Oct 23, 2023
85df54d
Change method signature
lauraporta Oct 23, 2023
0921dd9
Rearrange checks logic
lauraporta Oct 23, 2023
f21e252
Add useful counts
lauraporta Oct 23, 2023
6e13bdd
Extend adjust rotation increment
lauraporta Oct 23, 2023
c8a4a87
Remove obsolete method
lauraporta Oct 23, 2023
49b1a70
Add further method to correct ticks number
lauraporta Oct 23, 2023
b5edcce
Add better angle interpolation method
lauraporta Oct 23, 2023
23e9549
Use new interpolation method
lauraporta Oct 24, 2023
d9eba64
Change order of methods
lauraporta Oct 24, 2023
c08181e
Improve mask generation
lauraporta Oct 24, 2023
e0f84df
Read speed info as well
lauraporta Oct 24, 2023
2b58231
Fix interpolation bugs
lauraporta Oct 24, 2023
dfc69fd
Add converters
lauraporta Oct 24, 2023
98ee791
Add more debugging plots
lauraporta Oct 24, 2023
ae1fe1b
Reordering methods
lauraporta Oct 24, 2023
71ec4da
Improve calculation of rotation increments
lauraporta Oct 24, 2023
11faee0
Add typing and docstrings
lauraporta Oct 24, 2023
f513e48
Add first working test
lauraporta Oct 24, 2023
3800fc5
Add test on correction of start and end times
lauraporta Oct 24, 2023
1e61d01
Add test for the creation of the signed rotation array
lauraporta Oct 24, 2023
fd60648
Make drop ticks method static and create related test
lauraporta Oct 24, 2023
b738a13
Update manifest file
lauraporta Oct 24, 2023
3c00cd1
Update docstring
lauraporta Oct 25, 2023
80806ca
Fix random seed bug
lauraporta Oct 25, 2023
36f37b5
Make adjust rotation increment static and add test
lauraporta Oct 25, 2023
779f75e
Remove over-use of static methods, just one was useful
lauraporta Oct 25, 2023
6405684
Fix tick drop bug and add more tests
lauraporta Oct 25, 2023
a9329aa
Add test for angles
lauraporta Oct 25, 2023
e50cea3
Add new visualization and clean up
lauraporta Oct 25, 2023
2e20429
Add more docs and edit config
lauraporta Oct 25, 2023
eeb2af2
Add more docs
lauraporta Oct 25, 2023
3a8bcd2
Refactor and clean up
lauraporta Oct 26, 2023
e09fa49
Add working pipeline for incremental rotations
lauraporta Oct 26, 2023
8ca170c
Add saving of angle table, BUT there is a bug in the derotation, solu…
lauraporta Oct 27, 2023
66d9a05
Add working derotation of incremental images
lauraporta Oct 27, 2023
f7c6333
Add script to identify the displacement of the frames during rotation
lauraporta Oct 27, 2023
58d2344
Finish up script to test incremental derotation registration
lauraporta Oct 30, 2023
eea8e27
Add incremental derotation registration to the pipeline
lauraporta Oct 30, 2023
d92e92c
Allow to change diameter of circular mask
lauraporta Oct 30, 2023
430eda6
Fix derotation by line bug
lauraporta Nov 1, 2023
affee14
Delete all exploration plots
lauraporta Nov 2, 2023
9b6ea89
Delete napari related files
lauraporta Nov 2, 2023
affe179
Add typing and docstrings
lauraporta Nov 2, 2023
c666f15
Move scripts in a dedicated folder
lauraporta Nov 2, 2023
e1556ac
Remove leftover from napari plugin
lauraporta Nov 2, 2023
61ccb1e
Fix error in installation of dependencies
lauraporta Nov 2, 2023
3070708
Simplify derotation by line
lauraporta Nov 3, 2023
c8f3a8f
Add regression tests for image rotation
lauraporta Nov 3, 2023
ef75ccf
Update readme file
lauraporta Nov 3, 2023
866a4de
Expand readme
lauraporta Nov 3, 2023
91bbd8d
Add generic paths to config files
lauraporta Nov 3, 2023
a580964
Remove not so useful comments
lauraporta Nov 3, 2023
332d6dc
Update .gitignore and MANIFEST.in
lauraporta Nov 3, 2023
ee1c87c
Adapt sphinx docs and improve some docstrings
lauraporta Nov 6, 2023
85b04a9
Renaming
lauraporta Nov 10, 2023
b61c585
Move the main method, derotate by line, out of the pipeline
lauraporta Nov 10, 2023
e27e694
Add more examples in the readme
lauraporta Nov 10, 2023
54c80e4
Fix used dog image
lauraporta Nov 10, 2023
a261500
Add script to correct csv
lauraporta Nov 27, 2023
2bca2f7
Move the logic to generate a better csv inside the class
lauraporta Nov 28, 2023
c0ebc0c
Fixed manifest
lauraporta Nov 28, 2023
8d1f7db
Fix docs building bug (missing imports)
lauraporta Nov 28, 2023
7cade60
Add script to run derotation in the cluster
lauraporta Dec 11, 2023
59f7b45
Add the dogs 🐕
lauraporta Dec 12, 2023
1192ef1
Merge commit '7cade60719eb972594b2ea0644a9667cb8c1ad2f' into code-cle…
lauraporta Dec 12, 2023
1004279
Fix image naming mistake
lauraporta Dec 12, 2023
9ba7c3e
Fix bug in angle interpolation
lauraporta Dec 12, 2023
5731b04
Include contrast enhancment
lauraporta Dec 12, 2023
a73687b
Move threshold and contrast values to config file
lauraporta Dec 12, 2023
e63145c
Use the image offset to fill in the blank pixels
lauraporta Dec 12, 2023
6702fef
Fix typo
lauraporta Dec 12, 2023
418f339
Use the same diameter for masking when doing derotations one after th…
lauraporta Dec 12, 2023
b746f10
Update derotation/analysis/full_rotation_pipeline.py
lauraporta Dec 14, 2023
de94082
Update derotation/analysis/full_rotation_pipeline.py
lauraporta Dec 14, 2023
f50a681
Refactor example
lauraporta Dec 19, 2023
a9e5474
Improve usage of custom methods with docstrings and typing
lauraporta Dec 19, 2023
4f5e927
Substitute np.where with np.nonzero when only the condition is presen…
lauraporta Dec 19, 2023
8ff4bc4
Fix bugs found in the incremental rotation pipeline
lauraporta Dec 19, 2023
60853e5
Fix typo
lauraporta Dec 19, 2023
92dbd36
Fix creation of dataframe
lauraporta Dec 19, 2023
c7495f4
Rename variable k
lauraporta Dec 19, 2023
fbcf96c
Add missing dependencies
lauraporta Dec 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add functioning napari plugin
lauraporta committed Jul 10, 2023
commit b5a6acb8d91f4db78dac186438e83776d3757bae
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -26,3 +26,4 @@ repos:
- id: mypy
additional_dependencies:
- types-setuptools
- types-PyYAML
11 changes: 6 additions & 5 deletions derotation/adjust_rotation_degrees.py
Original file line number Diff line number Diff line change
@@ -4,17 +4,18 @@

import numpy as np
import scipy.optimize as opt
from find_centroid import (
from matplotlib import pyplot as plt
from scipy.ndimage import rotate

from derotation.find_centroid import (
detect_blobs,
extract_blob_centers,
find_centroid_pipeline,
get_optimized_centroid_location,
in_region,
not_center_of_image,
pipeline,
preprocess_image,
)
from matplotlib import pyplot as plt
from scipy.ndimage import rotate


def find_rotation_blocks(image_rotation_degree_per_frame):
@@ -175,7 +176,7 @@ def f(parameters):
diff_y = []
for k, img in enumerate(rotated_images):
try:
centers_rotated_image = pipeline(
centers_rotated_image = find_centroid_pipeline(
img, optimized_parameters[k]
)
center_dim_blob = mean_x, mean_y
3 changes: 2 additions & 1 deletion derotation/analog_preprocessing.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import copy

import numpy as np
from optimizers import find_best_k

from derotation.optimizers import find_best_k


def get_missing_frames(frame_clock):
77 changes: 77 additions & 0 deletions derotation/derotation_pipeline.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
from scipy.signal import find_peaks

from derotation.analog_preprocessing import (
apply_rotation_direction,
check_number_of_rotations,
find_rotation_for_each_frame_from_motor,
get_missing_frames,
get_starting_and_ending_frames,
when_is_rotation_on,
)
from derotation.find_centroid import find_centroid_pipeline
from derotation.get_data import get_data


class DerotationPipeline:
def __init__(self, path):
(
self.image,
self.frame_clock,
self.line_clock,
self.full_rotation,
self.rotation_ticks,
self.dt,
self.config,
self.direction,
) = get_data(path)
self.rot_deg = 360

print("Data loaded")

def process_analog_signals(self):
self.missing_frames, self.diffs = get_missing_frames(self.frame_clock)
(
self.frames_start,
self.frames_end,
self.threshold,
) = get_starting_and_ending_frames(self.frame_clock, self.image)
# find the peaks of the rot_tick2 signal
rotation_ticks_peaks = find_peaks(
self.rotation_ticks,
height=4,
distance=20,
)[0]

check_number_of_rotations(
rotation_ticks_peaks, self.direction, self.rot_deg, self.dt
)
self.rotation_on = when_is_rotation_on(self.full_rotation)
self.rotation_on = apply_rotation_direction(
self.rotation_on, self.direction
)
(
self.image_rotation_degree_per_frame,
self.signed_rotation_degrees,
) = find_rotation_for_each_frame_from_motor(
self.frame_clock,
rotation_ticks_peaks,
self.rotation_on,
self.frames_start,
)

print("Analog signals processed")

def calculate_centers(self, img):
lower_threshold = -2700
higher_threshold = -2600
binary_threshold = 32
sigma = 2.5

defoulting_parameters = [
lower_threshold,
higher_threshold,
binary_threshold,
sigma,
]

return find_centroid_pipeline(img, defoulting_parameters)
6 changes: 3 additions & 3 deletions derotation/find_centroid.py
Original file line number Diff line number Diff line change
@@ -40,7 +40,7 @@ def extract_blob_centers(properties):
return centroids


def pipeline(image, x):
def find_centroid_pipeline(image, x):
lower_threshold, higher_threshold, binary_threshold, sigma = x
img = preprocess_image(image, lower_threshold, higher_threshold)
labels, properties = detect_blobs(img, sigma, binary_threshold)
@@ -66,7 +66,7 @@ def cb(xk):
)

def f(x):
centroids = pipeline(image, x)
centroids = find_centroid_pipeline(image, x)
count_valid_centroids = 0
for c in centroids:
if not_center_of_image(c) and in_region(c):
@@ -86,6 +86,6 @@ def f(x):
)

# now find the centroid
centroids = pipeline(image, res.x)
centroids = find_centroid_pipeline(image, res.x)

return centroids, res.x
3 changes: 2 additions & 1 deletion derotation/get_data.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
import numpy as np
import tifffile as tiff
from read_binary import read_rc2_bin
from scipy.io import loadmat

from derotation.read_binary import read_rc2_bin


def get_paths(path):
path_tif = path / "imaging/runtest_00001.tif"
59 changes: 48 additions & 11 deletions derotation/plotting.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
import matplotlib.pyplot as plt
from pathlib import Path
from typing import Optional

import numpy as np
from napari.viewer import Viewer
from napari_matplotlib.base import SingleAxesWidget
from qtpy.QtWidgets import (
QPushButton,
QVBoxLayout,
QWidget,
)

from napari.viewer import Viewer
from typing import Optional
from derotation.derotation_pipeline import DerotationPipeline
from derotation.find_centroid import in_region, not_center_of_image

from napari_matplotlib.base import SingleAxesWidget
import numpy as np

class DerotationCanvas(SingleAxesWidget):
def __init__(
@@ -19,11 +21,11 @@ def __init__(
parent: Optional[QWidget] = None,
):
super().__init__(napari_viewer, parent=parent)
self.angles_over_time = np.sin(np.linspace(0,10,100))
self.angles_over_time = np.zeros(100)
self._update_layers(None)


def draw(self):
self.axes.clear()
self.axes.plot(self.angles_over_time, color="red")
self.axes.set_title(f"z={self.current_z}")
self.axes.axvline(self.current_z)
@@ -34,10 +36,45 @@ def __init__(self, napari_viewer: Viewer):
super().__init__()

self._viewer = napari_viewer

my_path = Path("/Users/laura/data/230327_pollen")
self.pipeline = DerotationPipeline(my_path)
self._viewer.add_image(
self.pipeline.image, name="image", colormap="turbo"
)
self.setLayout(QVBoxLayout())
self.button = QPushButton()
self.button.setText("Make plot")
self.layout().addWidget(self.button)

self.analyze_button = QPushButton()
self.analyze_button.setText("Run analysis")
self.analyze_button.clicked.connect(self.analysis)
self.layout().addWidget(self.analyze_button)

self.mpl_widget = DerotationCanvas(self._viewer)
self.layout().addWidget(self.mpl_widget)
self.layout().addWidget(self.mpl_widget)

def analysis(self):
self.pipeline.process_analog_signals()

self.mpl_widget.angles_over_time = (
self.pipeline.image_rotation_degree_per_frame
)

self.mpl_widget.draw()

correct_centers = []

for i in range(len(self.pipeline.image)):
centers = self.pipeline.calculate_centers(self.pipeline.image[i])
# print(centers)
for c in centers:
if not_center_of_image(c) and in_region(c):
correct_centers.append(c)
continue

centers = [
[t, coord[0], coord[1]] for t, coord in enumerate(correct_centers)
]

self._viewer.add_points(
centers,
)
11 changes: 7 additions & 4 deletions derotation/rotate_images.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import numpy as np
from find_centroid import pipeline
from scipy.ndimage import rotate

from derotation.find_centroid import find_centroid_pipeline


def rotate_images(
image,
@@ -37,12 +38,14 @@ def rotate_images(
# params = optimized_parameters[i]
# if i in indexes else defoulting_parameters

centers.append(pipeline(image[i], defoulting_parameters))
centers.append(find_centroid_pipeline(image[i], defoulting_parameters))
centers_rotated.append(
pipeline(rotated_image[i], defoulting_parameters)
find_centroid_pipeline(rotated_image[i], defoulting_parameters)
)
centers_rotated_corrected.append(
pipeline(rotated_image_corrected[i], defoulting_parameters)
find_centroid_pipeline(
rotated_image_corrected[i], defoulting_parameters
)
)

return (