post call 20020520

meeting-notes/author-session-CI2024-artifact-evaluation.md

@@ -5,8 +5,8 @@ description: Artifact Evaluation Author Q&A
 
 :::info
 **Live notes on hackmd:** https://hackmd.io/@turing-es/B1G8yeHmC/edit
-**Archived notes on github**: 
-**Call recording:** ==[...]==
+**Archived notes on github**: https://github.com/alan-turing-institute/climate-informatics-2024-ae/blob/main/meeting-notes/author-session-CI2024-artifact-evaluation.md
+**Call recording:** ==To be uploaded to CI/Turing youtube==
 
 :::info
 :::spoiler **Expand for Contents**
@@ -26,20 +26,97 @@ description: Artifact Evaluation Author Q&A
 ## Attendees
 (Name / Primary affiliation / Pronouns)
 
--
+- Alejandro Coca Castro (Turing, he/him)
+- Marion Weinzierl (ICCS, University of Cambridge / she/her)
+- Dominic Orchard (ICCS, University of Cambridge, he/him)
+- Hibrod Assa
+- Andrew Hyde (Cambridge University Press)
+- Lelle Demertzi (Turing, she/her)
+- Damon Wischik (Cambridge university)
+- Roly Perera
+- Oscar Key (University College London, he/him)
+- Douglas Rao (North Carolina Institute for Climate Studies)
+- 
 
 
 ## Notes
 
--
+- Introduction to the team
+* Resources
+    * [Github repository](https://github.com/alan-turing-institute/climate-informatics-2024-ae)
+    * [Reproducibility Guide in Turing Way Handbook](https://the-turing-way.netlify.app/reproducible-research/reproducible-research)
+    * [Reusable Code in The Turing Way](https://the-turing-way.netlify.app/reproducible-research/code-reuse)
+    * [Documentation on the process](https://github.com/alan-turing-institute/climate-informatics-2024-ae)
+    * [ACM Badges our badges are based on](https://www.acm.org/publications/policies/artifact-review-and-badging-current)
 
+### Context
+Reproducibility crisis: many scientific studies cannot be replicated or reproduced (especially in psychology and health research). This has mobilised the open source movement
+
+![Screenshot (57)](https://hackmd.io/_uploads/BkFu-ju7C.png)
+
+AE will benefit not only author's research, but also the community as a whole.
+
+**Climate Informatics**: A computational community in climate science. Annual Hackathons next to the conference, outputs published at EDS Book.
+
+### Artifact Evaluation
+
+Peer-review process of the paper: 
+* Submission phase for software artifact supporting the paper
+* Two reviewers will assess whether it meets AE standards
+* Addendum published with the CUP paper
+
+Why? 
+
+Promote reporducibility
+Promote increased impact in open source science
+Reviewers will co-author perspective paper on AE process
+
+Badges:
+[Available] : Publically accessible DOI/ Github repo with open source license
+[Functional] : Documented and complete, can be used, consistent with publication
+[Reusable] : Exceeds minimal functionality, facilitates reuse and repurposing, well-documented  and -structured
+
+### Logistics - Timeline
+What to submit?
+1. Overview of artifact
+    * Introduction
+    * Hardware requirements
+    * Getting started guide
+    * Step-by-step instructions
+    * Reusability guide
+3. Non-institutional URL pointing to: A) Single file b) Public repository
+    * Non-institutional URL essential for anonymisation with reviewers
+5. A hash certifying the version of the artifact
+
+Evaluation over 5 weeks
+
+![Screenshot (58)](https://hackmd.io/_uploads/SkwrVidmR.png)
+Submission - Conversation - Implementation - Scoring - Badge
+
+### EDS Publication
+CI2024 authors have the possibility to publish at EDS Journal. 
+Articles will be published in July 2024, with Data Availability Statement. 
+After AE (Sep - Oct 2024), addendum 'Artifact Evaluation Award' published. Addendum will be linkable and citable.
+![Screenshot (59)](https://hackmd.io/_uploads/S1bOHo_XR.png)
+![Screenshot (60)](https://hackmd.io/_uploads/ByDuSoOQ0.png)
+
+**Platform HotCRP (anonymous conversation)
+Deadline: 12th September, 2024**
+Comms:
+* Slack #ci-2024-ae
+* Github (link above)
+* Twitter @Clinformatics
 
 
 ## Questions
 (please add your questions here)
 
--
-
+- Why do you want a "non-instutional URL"? I'd have thought that there are institutional URLs (tech reports etc.) that are far more persistent than fly-by-night cloud services!
+    - Answered in comments above. -> anonymisation with reviewers
+- Software used for papers can also be used for consultancy. In that case, it might not be benefitial to share the software openly. We need protection of the things that we share in this case.
+    - Answer: Depends on the nature of what you have. If all of the code is not sharable it would be difficult to go for the "Available" badge, but you might be able to go for the "Functional" badge if you can share your code with the reviewers (otherwise it won't work). "Reusable" might or might not be possible in that case (by whom would the software be reusable?). Think creatively (use proxys, e.g.) -> discuss with reproducibility chairs.
+- Reproducibility vs generalisable -> is generalisable not more relevant for science? 
+    - Adapting [ACM work](https://www.acm.org/publications/policies/artifact-review-and-badging-current), where there are further badges. Work involved to go up to the "higher tiers" is massive, therefore we start with the three we mention. We are still really far from a situation where more is easily possible. -> Should also publish negative results. -> Might extend in future years.