Evaluate and get feedback from users on SDR alpha

[llivermore]

The production SDR instance at the stage where we can conduct user testing (following on from #60 ).

There are separate focused issues for tool performance, but we want to check that users can do the following testing tasks:

1. Login
2. Run a shared workflow
3. View outputs and download data
4. Copy and edit the “PinnedInsect-TenRow-Example.csv”
5. Repeat steps 3 and 4 using your edited specimen file and make some notes on how good the regions of interest are

Verbose instructions to follow....

[llivermore]

I created multiple testing account for some of the SDR Project Team members (and happy to add more as required).

Task Overview

1. Login
2. Run a shared workflow
3. View outputs and download data
4. Copy and edit the “PinnedInsect-TenRow-Example.csv”
5. Repeat steps 2 and 3 using your edited specimen file and make some notes on how good the regions of interest are

Run a shared workflow

1. Download “PinnedInsect-TenRow-Example.csv” and move it somewhere accessible
2. On the righthand side click on “Unnamed history” and rename it something sensible (like “My first workflow test”) so you have some context – consider a workflow like a mini project for a submitted dataset or testing
3. Click on "Shared Data" link at the top of the screen
4. Click on the “Workflow” link on the dropdown menu
5. Click on down arrow next to "DLA-Collections-test"
6. Click on the grey folder button to the right of long field box .
7. Click “Upload”
8. Click “Choose local files”
9. Select “PinnedInsect-TenRow-Example.csv” from wherever you moved it
10. Click “Start” and wait for the status bar to go to 100%
11. Click “Select” – you should see the dataset appear in your history with a green background
12. Click “Run Workflow”
13. Wait for a few minutes – the workflow will run through all of the steps (and substeps), each of which should go from grey (waiting) to peach? (running/in-progress) to green (complete)

Viewing outputs and downloading data

1. Click on (what should be) Step 48 in your history once the workflow has run – it should have a title like “SDR Teklia worker-dla on collection 37 and collection 36”
2. Click on one of the split_files – this should expand and give you more information
3. Click on the tiny graph button with hover over label “Visualise this button” to the left of the “?” button
4. Click ROI Visualisation – this should show you a visualisation of the regions of interest (ROI)
5. Once you have finished looking at any of the split_files click on the link under the “History” heading “< Back to [your history name]”
6. Click on the last step of the workflow labelled “sdr_output” to expand the workflow step
7. Click on the tiny left most button with the floppy disk icon with hover over label “Download” – this should download a zip file containing a folder “Bundled_Collection”

Copy and edit the “PinnedInsect-TenRow-Example.csv”

1. Open file in Excel
2. Find some “standard pinned insect” images from any project – the recent micro macros would be great
3. Update the “Catalog number”, “ Image URI”, “Higher classification” fields (if you are feeling lazy or are short for time, just change the Image URIs – if you have the time, add in your name and ORCID)

Using the following test file: PinnedInsect-TenRow-Example.csv

[emhaston]

This instruction is not clear

I'm seeing this

[emhaston]

Clicked Run and got this

which I think gives me the folder icon in the next step

[emhaston]

Workflow completed but didn't get a Step 48. Got this output

[OliverWoolland]

Workflow completed but didn't get a Step 48. Got this output

Thanks for this! I'm curious if this continued when you changed page or if the history was refreshed? The workflow looks to have completed ok but in the screenshot the history seems to lag behind the main report

[matdillen]

What is the vocabulary (if it's used by the workflow) of the Object type column, beyond "Pinned insect"?

I managed to run 5 herbarium sheet images from Zenodo through the DLA workflow. Everything went smooth and the bug outputting the wrong output (filename instead of JSON response) is indeed fixed. The results of the segmentation are not that great, which is to be expected given the different object type.

During my first run on the sample set provided by Laurence, connecting to the file upload server took a few minutes (as stated in the beginning of today's meeting).

One minor UI issue is that on the ROI visualization the metadata of boxes are always shown at the top of the page, so they cannot be seen for boxes that are at the bottom of an image with high height to width ratio. I wonder if it can be easily tweaked to show this metadata popup at the location of the box instead?

[Cubey0]

We (@emhaston & Robyn Drinkwater ) at the RBGE have had a test and we agree with @matdillen that the segmentation model being used may well be an issue preventing a proper assessment of the output.

On top of that @Cubey0 is having an error

When trying to upload an RBGE Image (via the individual file upload or via the URL upload)

It might well be due to a list being empty.

[llivermore]

What is the vocabulary (if it's used by the workflow) of the Object type column, beyond "Pinned insect"?

@matdillen it is not used by the workflow, although was considered a mandatory input field as we expected we may use it to select which model(s) to use if they were object-specific. I will need to double check the terms, but I expect they are:

1. Pinned insect
2. Herbarium sheet
3. Microscope slide

Note that the segmentation model in that workflow has only been trained on pinned insects (see #96 ) but the line detection segmentation had been trained on pinned insects AND herbarium sheets.

[llivermore]

We (emhaston & Robyn Drinkwater ) at the RBGE have had a test and we agree with matdillen that the segmentation model being used may well be an issue preventing a proper assessment of the output.

@Cubey0 as per previous comment, note that we only have a segmentation model for pinned insect images at the moment.

On top of that Cubey0 is having an error. When trying to upload an RBGE Image (via the individual file upload or via the URL upload)

@Cubey0 can you share the CSV file you were using to upload?

[Cubey0]

I didn't upload via a csv but pasted a set of image URLs , uploading went ok (as far as I could tell - I could see big jpg's in the workbench).

Also uploaded individual image and got the same error.

I will make a csv (mimic of your one) and see if get the same issue.

R

[OliverWoolland]

I didn't upload via a csv but pasted a set of image URLs , uploading went ok (as far as I could tell - I could see big jpg's in the workbench).

This is interesting - it is not a use case we have provided functionality for (to my knowledge). At the moment I think only csv upload is viable using existing tools.

There have been discussions about making it easier to drive individual tools using more friendly inputs (such as an actual image) but at the minute the tools are all driven using openDS objects which are somewhat cumbersome to create by hand.

[infinite-dao]

I did run successfully the imported: DLA-Collections-test on PinnedInsect-SingleRow-Example.csv.

Doing it in the same manner and run it on an own herbarium data failed however. (see the following sample)
JACQ-Herbarium-Sheet-Examples.csv

It seems similar to #100 (comment) and here the report:

I also found a glitch when pressing on history step 30: sdr_output at the view data, it seems that CSS is there that should not be at this place (developer console revealed it is within the <iframe> … <body>CSS-mix…</body></iframe>)

[OliverWoolland]

Doing it in the same manner and run it on an own herbarium data failed however. (see the following sample)

Thank you for providing the file! I tried running it and I can reproduce your error - I'll dig into it!

[Cubey0]

So I tried the Workflow: HTR-Collections-test with a large dataset (200 item herbarium specimens - the gold standard dataset from Edinburgh) and got some odd errors.

Also what are the results on the RHS - I'm seeing JSON data but not sure what the difference bits are

Some split file errors

But 200 ran through OK (with the Split errors)

Rob