Evaluation, Reproducibility, Benchmarks Meeting 27

Minutes of meeting 27

Date: 24th July 2024

Present: Anne, Annika, Carole, Keyvan, Lena, Michela, Nick, Nicola, Olivier

TOP 1: Continuation of the WG

• Election of chairs and secretary
  • Annika and Carole were elected unanimously as future WG chairs by the members of the WG
  • Nick was elected unanimously as future WG secretary
  • It was decided that the position will be for 2 years
  • Lena will get back to Michael Baumgartner to see if their group have any vetos
• Requirement: The WG also has to contribute code
  • Carole agreed to take over this role and will attend the meetings of the MONAI developers
• Proposal to integrate continuous MONAI segmentation benchmark as proposed by Michael Baumgartner
  • The WG agreed that we can contribute in terms of conceptualization of the benchmark and frame and also in the implementation of theoretical measures (such as metrics and aggregation)
  • The WG agreed that we should neither take over the full implementation of the framework nor the maintenance of the benchmark
    • For that, there should be a separate task force which will be advised by this WG
    • Michela stated interest in contributing more on this topic

TOP 2: Metrics Reloaded Software paper

• The WG decided to keep the package as standalone, but we will test if it is easily accessible both within and outside of MONAI
• Proper community testing of the full pipeline is still needed:
  • This will be done by Annika’s/Lena’s group as well as from members of Olivier’s group in September
  • Afterwards, the tutorials will be finalized during a hackathon at UCL in November
  • Carole and Annika will prepare the testing requirements
• Extensions:
  • New metrics (synthesis/surgery)
  • Variability, confidence
  • Benchmarking visualization (rankings reloaded)
• Potential journals:
  • SoftwareX
  • JMLR Software track
  • IEEE Transactions on Software Engineering (TSE)
    • Impact Factor: 3.514 (2022)
    • Focuses on all aspects of software engineering, including theory, methods, tools, and empirical studies.
  • ACM Transactions on Software Engineering and Methodology (TOSEM)
    • Impact Factor: 2.738 (2022)
    • Covers research on software engineering methodologies, techniques, and tools.
  • Journal of Systems and Software (JSS)
    • Impact Factor: 2.829 (2022)
    • Publishes research on software development, maintenance, and management.
  • Software: Practice and Experience (SPE)
    • Impact Factor: 1.731 (2022)
    • Focuses on practical and experimental research in software development.
  • Empirical Software Engineering (EMSE)
    • Impact Factor: 2.662 (2022)
    • Emphasizes empirical studies in software engineering.
  • Automated Software Engineering (ASE)
    • Impact Factor: 2.102 (2022)
    • Concentrates on automated methods for software engineering.