Dredge Motion correction gets stuck on Solving

[kkatlowitz]

Hello,
I am trying to use dredge though SpikingInterface. It does detect and localize, cross correlation
but then gets stuck (0/100%) on the solve portion. It can run just fine on segments of the data but when I give it more than 20 minutes it breaks

Its a fresh install using the .yml file
Windows 11
Python 3.11.10
I have a GPU using CUDA

job_kwargs = dict(n_jobs=40, chunk_duration="1s", progress_bar=True)
preset_keys = get_motion_presets()
one_preset_params = get_motion_parameters_preset("dredge_fast")
base_folder = Path('../Record Node 108/experiment1/')
preprocessed_folder = base_folder / 'preprocess'
rec = si.load_extractor(preprocessed_folder)
folder = base_folder / "motionCorrTest"
recording_corrected, motion, motion_info = si.correct_motion(rec, preset="dredge", folder=folder, output_motion=True, output_motion_info=True, **job_kwargs)

[kkatlowitz]

Some other details on the failure mode:
I tried blanking the data (replacing it with zeros) at the segment that it gets stuck at, didnt help.
It doesnt always get stuck at solve 0%, sometimes it gets stuck at solve 1/8 or 7/8
to fix this problem I just do

rec1 = rec.frame_slice(start_frame=0, end_frame=21*60*30000-1)
rec1D, motion1, motion_info1 = si.correct_motion(rec1, preset="dredge_fast", folder=folder, output_motion=True, overwrite=True,output_motion_info=True, **job_kwargs)
rec2 = rec.frame_slice(start_frame=21*60*30000, end_frame=None)
rec2D, motion2, motion_info2 = si.correct_motion(rec2, preset="dredge_fast", folder=folder, output_motion=True, overwrite=True,output_motion_info=True, **job_kwargs)
concatenated_recording = si.concatenate_recordings([rec1D,rec2D])

[zm711]

@cwindolf @samuelgarcia

[cwindolf]

Hi @kkatlowitz , thanks for the question (and thanks @zm711). I haven't seen it get stuck in the solve part before. I would say that problems of this size (<1hr) are typical cases that it should handle fine, so I'm interested to narrow this one down.

I'm trying to think of a way to figure out why / where it's getting stuck. I think one method would be: if you interrupt your Jupyter notebook kernel after it's been stuck for a while, we should be able to see what line of code it was running. (Or, sometimes the exception will raise at the next line of code.)

My best guess would be that it's the solver that is getting stuck? I think that's the most expensive line of code... but maybe it's better to check what the output of the above check is before testing that?

[kkatlowitz]

I appreciate the incredibly fast response. I tried interrupting the kernel, but it wouldnt stop. I had to restart to kernel to get it to quit.
I am running
spikeinterface 0.101.2
The CPU on my task manager is pretty full at the time like 70% of a 5GHz processor

While I have you on this topic: motion_info gives the displacement at 1Hz. Is it possible to get it to output the finer resolution information? I think Dredge works at like 250Hz or something like that.

[kkatlowitz]

I tried it on a diff dataset and it also got stuck. also solved by making it shorter
This is neuropixel (384 channel) data

[samuelgarcia]

Hi.
Did you try to force torch to use the cpu instead cuda ? You can use estimate_motion(...., method="dredge_ap", ..., device="cpu", ...).

[kkatlowitz]

I had to lower the chunk_duration to 0.5 from 1s (it kept giving me out of memory issues) but it worked.
this is workable but Id rather do GPU if possible.

job_kwargs = dict(n_jobs=40, chunk_duration="1s", progress_bar=True)
rec, motion, motion_info= si.correct_motion(rec, preset="dredge_fast", folder=None, output_motion=True,output_motion_info=True,  
                                            estimate_motion_kwargs=dict(
                                                    device="cpu",
                                                    bin_s=0.1,
                                            ),
                                            **job_kwargs,)