Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Video Cropping Pipeline (General Purpose) #3

Open
6 tasks
jmdelahanty opened this issue Feb 1, 2022 · 4 comments
Open
6 tasks

Video Cropping Pipeline (General Purpose) #3

jmdelahanty opened this issue Feb 1, 2022 · 4 comments

Comments

@jmdelahanty
Copy link
Contributor

@rpamintu and @hadiviko have expressed interest in making a way forward for cropping videos in a flexible way for use in AlphaTracker pipelines. There's currently a Jupyter Notebook that's been written by a former lab member, Shristi, for @carolinejia22 and uses moviepy for performing the cropping.

Unfortunately, there were inconsistencies in how recordings were performed (which will be corrected in the future). Thus, many of the recordings have inconsistent locations inside the video frame for home cage sociability recordings. We need a way of cropping this sort of data in a reproducible, easy to use, and documented manner.

What can come of this project is creating a general purpose image cropping tool that can output correctly formatted single images or videos that the lab can use.

There are two options I see for moving forward here:

  1. MATLAB: MATLAB has an image cropping tool that can perform this task for us it seems. See this link. We can probably make something much quicker than we could in Python if we use this. The primary issue is that we would either need to get you all MATLAB installed on your laptops (expensive, no thanks) or you can hop onto the cluster to use MATLAB licenses there. This option is ideal, plus you can get familiar with using the cluster generally. It's not too scary, I promise!
  2. Python: Python's OpenCV library has many ways of performing this sort of cropping by letting you make apps that can draw over images, perform a crop for all images in a video, and then write it all to some kind of output. This will require much more work than the MATLAB option most likely and, therefore, might not be the preferred method of moving forward with this. However, on the other hand, it would offer the best opportunity to learn a lot more about image manipulation details than the solution MATLAB offers you.

Comment on this post (it's fun!) for what you think. Try reacting to the post too for voting: ❤️ for python, 🎉 for MATLAB.

Regardless of the option you choose, the following will need to be done:

REQUIRED

  • Where will this data be stored on the server? We need to have all videos have a consistent naming convention (ideally one that other teams in the lab could also use) and stored in a way that's structured, easy to programmatically write to and read from.
  • What should the outputs look like? Where will those go?
  • How should we keep track of the different things that have been completed for each step? This is something I've been grappling with and have probably been trying to make far too complicated. A simple spreadsheet for now would work fine, but ideally it could be checked by a script that anyone in the lab could quickly run to see what's been through what step without having to check directories themselves.

OPTIONAL BUT COOL

  • How do we make this a reliable, easy thing to use for anyone in the lab for any purpose?
  • Can we make a template for different use cases that can run different things in batches?

If you choose to do the Python version, there's a few other things we could do that would be great for you to learn about if you plan on doing any sort of larger than memory data processing (which if you do any sort of functional imaging like 2p/1p or even large ephys recordings, you will do one day! It's the future happening now!).

  • Use Zarr as a data storage format for video data (I can explain Zarrs later if you're interested)

Let me know what you think. Depending on how you vote we'll move from there...
@carolinejia22
Copy link

carolinejia22 commented Feb 1, 2022 via email

@jmdelahanty
Copy link
Contributor Author

Hello Caroline!

What we would do is probably what you said here. We'd define the crop by drawing a box around the object and use this as an ROI mask. We would then have to get every frame in the video, perform this processing on it, and then write a new video file. I'm pretty sure this is what moviepy does but it just does all that in the background for you so you don't have to write out that process yourself. The main thing we'd have to do for a minimally working Python version of this would be to define the ROIs ourself (the crop positions), using opencv's library and then feed those into moviepy. It's definitely possible to do that.

There may be a library in MATLAB that performs these processes for you on videos, but I didn't see one.

@jmdelahanty
Copy link
Contributor Author

I see @rpamintu has voted, what say you @hadiviko ?

In the meantime @carolinejia22 , Rachelle has apparently gotten VLC to successfully crop videos. She'll be documenting it so we can see what we were doing wrong with the VLC editor on your PC.

@carolinejia22
Copy link

carolinejia22 commented Feb 2, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jmdelahanty @carolinejia22