Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code to select the aod motion computation method based on file versio… #16

Open
wants to merge 30 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
8486b32
Volume ignore improvement
Jun 18, 2013
f250860
AOD: switch some objects to dj.AutoPopulate
peabody124 Dec 18, 2013
ee7d0f7
AOD: check in some changes laying around
peabody124 Dec 18, 2013
c1307c1
Merge branch 'master' of github.com:peabody124/sessions
peabody124 Dec 18, 2013
18c46c0
Merge branch 'master' of github.com:peabody124/sessions
Dec 18, 2013
a9aa2f1
Whatever James had not commited...
Jul 10, 2014
7e0835e
Fix bug where running new spike detection method deletes previous det…
eywalker Jan 12, 2016
45a1f9e
SyncNetworkFewTrials: check in missing file
eywalker Jun 4, 2016
a61d775
AOD: switch some objects to dj.AutoPopulate
peabody124 Dec 18, 2013
7e2dc49
AOD: check in some changes laying around
peabody124 Dec 18, 2013
4ac02b1
Code to select the aod motion computation method based on file versio…
Feb 20, 2017
452df84
1. changed variables names
Feb 21, 2017
0a854db
Work remotely
peabody124 Mar 23, 2017
418afa1
Merge branch 'master' of https://github.com/saumilpatel/sessions into…
peabody124 Mar 23, 2017
1d741aa
Changed location of checkFilename from +acq to +aod
Mar 23, 2017
817e259
Check in code for Saumil's motion tracking
peabody124 Sep 24, 2017
ee19c89
detectLcdPhotodiodeFlips: get rid of external dependecy
peabody124 Sep 25, 2017
aa2dad5
Small fixes
eywalker Sep 25, 2017
a667013
AOD: improve robustness of syuncing
eywalker Sep 25, 2017
f1c2a70
Merge remote-tracking branch 'origin/james'
eywalker Sep 25, 2017
d7620db
Make RawPathMap updatable
eywalker Oct 30, 2017
205cdc4
Adjust diode offset accepted range
eywalker Oct 30, 2017
5c681f8
Add special handling for NatImExperiment end time
eywalker Oct 30, 2017
1302179
Merge pull request #17 from peabody124/james
eywalker Oct 30, 2017
666ddae
Merge pull request #14 from atlab/bugfix
saumilpatel Oct 30, 2017
e2d236e
Merge branch 'master' of https://github.com/atlab/sessions
eywalker Oct 30, 2017
33b7752
code to record notification sent to slack in database
saumilpatel Oct 9, 2018
0a7ed69
Merge branch 'master' of https://github.com/saumilpatel/sessions
saumilpatel Oct 9, 2018
f876949
added /Volumes/M to search path
saumilpatel Oct 9, 2018
a40ff29
NotificationSent table moved from acq to stimulation
saumilpatel Oct 11, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ getHost.m
.*
*~
~*
startup*.*
18 changes: 18 additions & 0 deletions schemas/+acq/checkFileversion.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@

function oldversion=checkFileVersion(fn)

oldversion = true ;
try
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is failing for me when analyzing

M/Mouse/2016-03-15_13-56-05/AODAcq/2016-03-15_14-38-45_%d.h5

specifically the h5readatt fails so we fall through to the catch statement. i believe this experiment was recorded with the new format, though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh... works fine for the file you mention on at-ssp, Matlab R2016b. Is h5readatt working for other files ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check it from a mac? i'm using this from my local machine (R2016a but that command was introduced in 2011a)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, h5readatt does not work on my Mac 2016a... let me check why

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it works if the file name is /Volumes/M....
Is this what you meant in the comment getFile... To translate the file name for the platform used ?

version = h5readatt(fn,'/','Version') ;
if (version==2)
oldversion = false ;
% motion data are in the new 3d format
else
br = aodReader(fn, 'Motion') ;
sz = size(br.motionCoordinates) ;
if rem(nthroot(sz(1),3),1)==0 % a way to handle files older than version 2 where a volume with cubic dimension was collected
oldversion = false ;
end
end
catch
end
26 changes: 17 additions & 9 deletions schemas/+aod/ScanMotion.m
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,24 @@ function makeTuples(self, key)

tuple = key;

br = getFile(acq.AodScan & key,'Motion');
motionData = br(:,:,:);
[T D P] = size(motionData);
mot = reshape(motionData,[T 10 20 P]);
mot = permute(mot, [4 2 3 1]);
t = br(:,'t');
coordinates = br.motionCoordinates;

fn = char(fetchn(acq.AodScan & key, 'aod_scan_filename')) ; % get the filename from db
oldversion = aod.checkFileversion(fn) ; % check if motion data is in old format, i.e. 2 planes or in new format, i.e. a volume
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a typo, or more likely the checkFileVersion should have been checked into the aod directory instead of the acq directory.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops... yes a typo, should be acq.checkFileVersion

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's probably best to keep with the aod schema since it doesn't apply to anything else in acq. Also, if we can refactor it into the AodMotionReader class so that this isn't exported in general that would be better (see comment below)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree... will move it

if (oldversion)
br = getFile(acq.AodScan & key,'Motion');
motionData = br(:,:,:);
[T D P] = size(motionData);
mot = reshape(motionData,[T 10 20 P]);
mot = permute(mot, [4 2 3 1]);
t = br(:,'t');
coordinates = br.motionCoordinates;


[xpos ypos zpos details] = aod.trackMotion(mot, t, coordinates);
[xpos,ypos,zpos,details] = aod.trackMotion(mot, t, coordinates);
else
% returned positions in microns and time in secs
% the motion data channel is hardcoded to 2, should be eventually guided by some info in the database
[xpos,ypos,zpos,t] = aod.trackMotion3D(fn, 2);
end
tuple.x = xpos;
tuple.y = ypos;
tuple.z = zpos;
Expand Down
86 changes: 86 additions & 0 deletions schemas/+aod/trackMotion3D.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@

function [x,y,z,t] = trackMotion3D(fn, analysischan)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we refactor this a bit so move all the code that deals with the raw file access into ScanMotion (like it is for the original code) and then make this method just take the data set and process it? that would keep the two consistent.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to run motion detection during realtime acq as well as during offline analysis. Because realtime implementation came first, the code is residing in folder structure that we have for realtime stuff. We can rearrange if it is absolutely necessary, for sure we dont want to duplicate the same code in different folders


%% test function computeshifts
% requires z:\libraries\matlab in path
run(getLocalPath('/lab/libraries/hdf5matlab/setPath.m')) ;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this shouldn't be here. the user should have set up the path correctly before hand. this would likely have unintended side effects.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree, will remove


downsampleFactor = 1 ; % skip frames if > 1
numFramesToAverageForRef = 10 ; % frames to average to determine the reference frame against which motion is computed
useMultiCore = true ;
postRefStartFrameTime = 0 ; % N secs to skip before analyzing motion signals, in many cases, right after starting the point scan, there is a slight drift

br = aodReader(fn, 'Motion') ;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here I think this should use getFile(fn, 'Motion') to convert the path from /raw/blah to the real location

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this I am not following...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ended up replacing this with

fn2 = findFile(RawPathMap, fn)
br = aodReader(fn2, 'Motion');
br1 = AodMotionCorrectionReader(fn2) ; % if motion was corrected by using aod offsets, this is where they would be read

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good

br1 = AodMotionCorrectionReader(fn) ; % if motion was corrected by using aod offsets, this is where they would be read
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need a separate class here just to access the sampling rate? can we not expand the original AodMotionReader to handle both formats correctly? I only see br1 used below to access the number of samples and sampling rate, which AodMotionReader should just be taught to do correctly for this new format.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not but we are just reusing code that we have.


dat = br(:,:,:) ;
numSamples = br1.sz ;
duration = numSamples(:,1)/br1.Fs ; % sec

motion_fs = size(dat,1)/duration ; % this is based on the temporal sampling rate
delta_t = downsampleFactor/motion_fs ;

numPlanes = br.planes ;
coordinates = br.motionCoordinates ;

% get pmt chan data
dat2 = squeeze(dat(:,:,analysischan)) ;
clearvars dat ;
gridSize = sqrt(size(dat2,2)/numPlanes) ;

% frame number to start from
postRefStartFrame = max(round(postRefStartFrameTime*motion_fs),1) ;

% chop off the first N secs
dat2 = dat2(postRefStartFrame:end, :) ;

needReference = true ; % first get a reference frame
refparams = [] ;
calibparams = [0 1; 0 1; 0 1] ; % dont apply any calibration
shifts = zeros(size(dat2,1),3); % store motion vectors here
dat2(1,:) = mean(dat2(1:numFramesToAverageForRef,:)) ; % average frames for computing reference
[shifts(1,:), refparams] = computeshifts(dat2(1,:)',coordinates,gridSize,calibparams,refparams,needReference) ; % this position of the fitted sphere is treated as the reference
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this function is missing. above it references it being in lab\libraries\matlab but i don't see it there. also, please let's not spread these dependencies across multiple repositories.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is in z:\libraries\TwoPhoton\AOD_Control\MotionCorrection. Note that we are using the same motion detection code that is used from the Labview program during real-time acquisition.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really like duplicating code in two places, but at the same time in this case I think it would be better than spanning dependencies between two pretty separate projects. It also avoids unintended side effects if one of the formats changes the other would break.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, would make the sessions folder self contained but we will have to remember to keep the two folders synced.


% setup for parallel processing
if (useMultiCore)
try
mypool = parpool ; % use max processors
catch
fprintf('Unable to start parallel pool, single processor will be used') ;
useMultiCore = false ;
end
end

% take into account frames to skip, i.e. sampling rate of motion,
% downsample with averaging
adat = zeros(size(dat2)) ;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replace these loops with matlab's implementation of downsampling.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did not change, cant find downsampling code that will average samples, dont want to deal with filters

if (downsampleFactor>1)
count = 0 ;
for ii=numFramesToAverageForRef+downsampleFactor:downsampleFactor:size(dat2,1)
count = count + 1 ;
adat(count,:) = adat(count,:) + sum(dat2(ii-downsampleFactor+1:ii,:)) ;
adat(count,:) = adat(count,:) / downsampleFactor ;
end
else
adat = dat2(numFramesToAverageForRef+1:end,:) ;
count = size(adat,1);
end

needReference=false ; % no need for a reference, start computing positions relative to the reference of spheres in subsequent motion frames
if (useMultiCore)
parfor ii=1:count
shifts(ii+numFramesToAverageForRef,:) = computeshifts(adat(ii,:)',coordinates,gridSize,calibparams,refparams,needReference) ;
end
delete(mypool) ;
else
for ii=1:count
shifts(ii+numFramesToAverageForRef,:) = computeshifts(adat(ii,:)',coordinates,gridSize,calibparams,refparams,needReference) ;
if (rem(count,100)==0)
fprintf('Processed %d frames', count) ;
end
end
end
x = squeeze(shifts(numFramesToAverageForRef+1:end,2)) ;
y = squeeze(shifts(numFramesToAverageForRef+1:end,1)) ;
z = squeeze(shifts(numFramesToAverageForRef+1:end,3)) ;
t=((0:1:length(x)-1)*delta_t)+(numFramesToAverageForRef/motion_fs) ; % the first motion position is after the last frame used for reference calculations