What metadata should we collect on models and datasources?

[edwardchalstrey1]

Now that we have http://sci.vision/ up and running, which displays some of the metadata for each model and datasource, the following questions have occurred to me:

1. Are there metadata fields we are missing?
   • Should models and datasources have an "author" or "provider"?
2. Should any of the fields we have be removed?
3. Any opinions on what fields should be shown or hidden when clicking on a model/ds in sci.vision web UI

There are also some proposals for new fields in #110

Currently for models, the fields are:

• name
• description
• tasks (multiple, should be from a pre-defined list but currently don't think there's anything stopping more being added)
• url (to GitHub)
• pkg_url (pip installable version of url)
• format (currently always "image")
• pretrained (bool, currently always true)
• labels_required (bool)
• institution (list)
• tags (list)

For datasources they are:

• name
• description
• tasks (same as above)
• domains (multiple, free text)
• url (to GitHub)
• format (currently always "image")
• labels_provided (bool)
• institution (list)
• tags (list)

[ots22]

See also #110 for some related ideas

[ots22]

Some thoughts on what we have currently

• Model

  • name
    • Currently these are all/mostly very short with no spaces - perhaps we could require this, as it makes certain uses more convenient. Or perhaps a 'short name' and a 'title'
  • description
    • I think this is fine, although most entries don't have descriptions
  • tasks (multiple, should be from a pre-defined list but currently don't think there's anything stopping more being added)
    • I think this is a list (meaning zero or more tasks), since the 'model' might conceivably have more than one predict endpoint, or the package contain several models. I think this is probably okay for the catalog, but would be useful to make sure this joins up with what is represented in model.yml. When we settle on some supported tasks we can make sure that these match, perhaps with a catch-all 'other' task label as well.
  • domains (suggested)
    • include this, similar to Datasource?
  • url (to GitHub)
  • pkg_url (pip installable version of url)
    • The python library does handle GitHub specially, but we should try with some that aren't hosted on GitHub! A separate repository URL and package seems reasonable
  • homepage_url (suggested)
    • This would be good for the frontend model card
  • format (currently always "image")
    • I think we can drop this (there might be a case for being able to specify particular requirements on the image, like number of channels)
  • pretrained (bool, currently always true)
    • I think we can drop this
  • labels_required (bool)
    • This information might be completely captured by the 'task' field, so consider dropping
  • institution (list)
    • There might not be an institution, or even an organisation (although perhaps likely in)
    • Some risk of redundant entries (e.g. 'The Alan Turing Institute' and 'Turing Institute' both showing up) which we should think about from a UI point of view
    • Think we should consider the uses we want this for before definitely keeping it
  • tags (list)
    • I see this as supporting keyword search

• Datasource

  • name
    • as above
  • description
    • as above
  • tasks (same as above)
    • Nothing this has a different meaning from a Model 'task' - it is the tasks that the contributor expects the dataset to be useful for
  • domains (multiple, free text)
    • relevant: Use standard keywords and phrases within the data and models catalogues #54 (comment)
  • url (to GitHub)
    • I think this can be to anywhere, as long as it points to a datasource yaml file, but with special support for GitHub
  • format (currently always "image")
    • drop (with same note as above)
  • labels_provided (bool)
    • This seems more useful than the Model 'labels_required', but maybe there is a way to capture this through tasks too
  • institution (list)
    • as above
  • tags (list)
    • as above

Having an author in a public catalog seems to have a few challenges. As soon as we start keeping author information, we have some personal data, with the various consequences of that. Authorship might change over time. The different entries might also represent their provider/author in subtly different ways (e.g. name/email address or perhaps a model only has a GitHub username). One possible way out is to link to some (external) citation information - a citation field storing a doi might be enough on its own, or a link to a CFF file if it exists - then someone else has already worried about this, and it is the upstream datasource/model's responsibility to keep this up to date.