Evaluation, Reproducibility, Benchmarks Meeting 30

Minutes of Meeting 30

Date: 27th November, 2024

Discussed whether we'd like to update our working group's name
- It might be nice to make the name shorter and more understandable
- Is evaluation + benchmarking redundant? Not necessarily. Benchmarking is more about the rigorous comparison between competing methods, whereas evaluation is more generally about assessing the usefulness of a system in a clinical setting.
- The consensus is "Evaluation and Benchmarking Working Group"
3 to 4 sentences describing the working group --
- Carole will add this blurb to Google Drive and invite comments/suggestions
  - If you have a different email you'd like to use for sharing folders/files via Google, please reach out to Nick at [email protected]

MONAI would like these working groups to be more accessible to and engaged with the public
Ideas for how we might go about this:
- Surveys to the public? We've had some experience with this, but didn't get much engagement
- MONAI days + related initiatives where we have a specific need have been very effective
- In general, we do want new, highly motivated members, but we don't want the group to get so big that it is unwieldy and inefficient
- New opportunities via the website
  - Make sure community is aware of current projects
  - Open up a "suggestion box" or other avenue for the community to indicate what they'd like us to work on
  - Need to talk to the team behind the website changes to see how much flexibility we have here

First one for the confidence intervals project
- To provide guidelines for reporting
- And practical implementation of tasks in Metrics Reloaded
- Olivier to take the lead
Second one about data
- What makes data suitable for benchmarking? -- related to dataset quality work in progress for MICCAI lighthouse challenges
- Working with the development team to integrate data downloads/loaders into MONAI itself
- Michela to take the lead
Overall structure
- Each task force to prepare some brief slides for WG meetings
- Alternate focus for each meeting
- Next meeting (January 22nd), we will play it by ear for which task force will be the focus

Copyright (c) MONAI Consortium