Training tracking request and introduction to ROV research using VIAME

[zhrandell]

Hello VIAME team,

I want to take this opportunity to (1) ask about a feature request, as well as (2) introduce you all to emerging Seattle Aquarium research using VIAME to analyze ROV data.

(1) Feature request: have you all made progress enabling pipelines we train ourselves to subsequently track individuals once the pipeline is applied to unseen imagery? As of now (as far as I’m aware based on using both the desktop and web versions), we can manually annotate individual tracks, though compiling and applying these pipelines produces detections where each frame is treated as “new”, such that repeat observations of the same individual are recorded as “new” detections. I can see that this tracking feature is supposed to be forthcoming (linked here under “Training”: “Tracker training will be added in a future update.”), and I wanted to check in regarding its status.

(2) I also wanted to share a bit more about how we’re using VIAME. Please see our GitHub repo, where we’ve organized information pertaining to our ROV/AI research. In short, we’re using small, maneuverable, customizable, and relatively inexpensive ROVs to survey within kelp forests with the broader objective of expanding the spatial extent of seafloor we can survey. You can see numerous ROV videos here, including a typical downward-facing video here, and a forward-facing video here. In the forward-facing video, note the numerous bull kelp stipes, as well as the occasional fish . . . these are the “objects” we wish to track across multiple frames (whereas we can analyze the downward-facing video with non-overlapping stills, negating the need to track individuals).

Thank you very much in advance for any information you may be able to provide. And thank you for supporting this amazing open-source software. It is our sincere hope these types of tools will change the way we analyze, make inferences, and conduct kelp forest conversation and restoration!

I’m at [email protected] if you ever want to discuss further.

All the best,

Zach

[waxlamp]

Hi Zach! Thanks for getting in touch!

(1) Feature request: have you all made progress enabling pipelines we train ourselves to subsequently track individuals once the pipeline is applied to unseen imagery? As of now (as far as I’m aware based on using both the desktop and web versions), we can manually annotate individual tracks, though compiling and applying these pipelines produces detections where each frame is treated as “new”, such that repeat observations of the same individual are recorded as “new” detections. I can see that this tracking feature is supposed to be forthcoming (linked here under “Training”: “Tracker training will be added in a future update.”), and I wanted to check in regarding its status.

While we haven't made explicit progress on the feature you mention, I believe we can help you out here; I'm going to defer to my colleague @mattdawkins to give you some idea of how we might do so.

(2) I also wanted to share a bit more about how we’re using VIAME. Please see our GitHub repo, where we’ve organized information pertaining to our ROV/AI research. In short, we’re using small, maneuverable, customizable, and relatively inexpensive ROVs to survey within kelp forests with the broader objective of expanding the spatial extent of seafloor we can survey. You can see numerous ROV videos here, including a typical downward-facing video here, and a forward-facing video here. In the forward-facing video, note the numerous bull kelp stipes, as well as the occasional fish . . . these are the “objects” we wish to track across multiple frames (whereas we can analyze the downward-facing video with non-overlapping stills, negating the need to track individuals).

Thank you very much in advance for any information you may be able to provide. And thank you for supporting this amazing open-source software. It is our sincere hope these types of tools will change the way we analyze, make inferences, and conduct kelp forest conversation and restoration!

I’m at [email protected] if you ever want to discuss further.

This is very cool stuff! And it seems quite aligned with what DIVE and VIAME have been able to provide for other folks in the past. I would love to learn more about what you're doing. Specifically, I'm interested in the scientific questions you're interested in, the data workflow you're using or hoping to use, and how we might be able to help out.

I'll be in touch via email to continue that discussion.

Thanks again for getting in contact! Excited to learn more!

[mattdawkins]

An existing (pre-trained) tracker can be added to trained detectors in the web or desktop clients with minimal effort, e.g. a minute or two. There's 3 different default trackers in the system right now for different purposes. One for high frame rate data with minimal changes between frames (e.g. lots of moving targets in 10hz+ data), one for low-frame rate data which only uses image registration on the background and assumes the targets don't vary too much frame to frame, and an intermediate of the two.

Assuming you're using the web version, to accomplish this, there's an advanced file manager at viame.kitware.com/girder, where you can see the pipeline configuration files that you've trained up and replace them or, for example, add a tracker. If you navigate to your user directory you should see all the models you've trained up in a VIAME/VIAME Training Results folder. In the Desktop version of DIVE this would be located in VIAME_DATA folder as specified in the settings panel. An example detector.pipe is attached, and also a detector.pipe with one of the default trackers added, which you can replace it with: replace original generated pipe file with the attatched (ie don't call it .txt I just named them here to get past githubs attatchment filter).

detector_orig.txt
detector_w_tracker.txt

Training up a new tracker model requires a little bit more effort, however, and that capability isn't presently in the GUIs (though is slated for eventual release). One of the tracker can be trained via example scripts in 'examples' folder of the desktop version, then the two other trackers require a small amount of python knowledge to be able to run python scripts to re-train them on new datasets. If interested in the scripts, ping me and I'll email them to you.

[mattdawkins]

Two out of three of these trackers can also be run as-is through the User Interface without modifying pipelines, with default models anyways, and they function slightly differently.

One is the Utilities->Link Detection Default will run the default fish multi-target tracker to link detections on individual frames into longer tracks. This method relies on having existing detections from a trained object detector on every frame, and it attempts to link those detections into tracks

The Utilities -> Track Selections Default Mask uses a different algorithm, one which just takes the first box on a target only and tries to track it, so you just want one box on the start of each target. It's meant more for annotation assist and uses a completely different algorithm than the Link Detections method