Automated Workflow Checks for Data and Models

[Tonks684]

Fix #479

[Tonks684]

I have created automated_dataset_checks.py file in the dev folder. As discussed with @ots22 and @acocac the file iterates through the dataset catalog using the load_dataset function and logs any datsets that fail to load. At the moment 2/11 fail and they are stardist and cell-cycle-scivision.

Cell Cycle Scivision
Error message: 404 File Not Found

Reason for error: Examining the load_dataset function it fails because the yaml file is '.yaml' and the path is hard coded to be '/scivision/data.yml'
Proposed solution: Remove hard coded '.yml' and allow for 'yml'

Stardist
Error message: 404 File Not Found

Reason for error: Examining the load_dataset function it fails because the url path is to a zip file.
Proposed solutions: Adjust the load_dataset function to work with zip files or we could create a .yml template for data uploads where the images are contained in a zip file.

[ots22]

Hi Sam, this seems very useful.

It looks like there are some additional commits to push (I can just see a single commit with a small change)?

If fixing the broken entries seems beyond the scope of this PR, definitely consider raising issues and we can address them separately.

[ots22]

@Tonks684 @acocac would you be happy with merging this, so the script is on main? Then making separate PRs for models/automation?

Are you happy too with the renaming (introduced above)?