Merge pull request #118 from rvandewater/submission_experiments

Major revision for release

CONTRIBUTING.MD

@@ -0,0 +1,101 @@
+# Contributing
+
+When contributing to this repository, please first discuss the change you wish to make via issue,
+email, or any other method with the owners of this repository before making a change. 
+
+Please note we have a code of conduct, please follow it in all your interactions with the project.
+
+## Pull Request Process
+
+1. Ensure any install or build dependencies are removed before the end of the layer when doing a 
+   build.
+2. Update the README.md with details of changes to the interface, this includes new environment 
+   variables, exposed ports, useful file locations and container parameters.
+3. Increase the version numbers in any examples files and the README.md to the new version that this
+   Pull Request would represent. The versioning scheme we use is [SemVer](http://semver.org/).
+4. You may merge the Pull Request in once you have the sign-off of two other developers, or if you 
+   do not have permission to do that, you may request the second reviewer to merge it for you.
+
+## Code of Conduct
+
+### Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+### Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+### Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+### Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+[//]: # (### Enforcement)
+
+[//]: # ()
+[//]: # (Instances of abusive, harassing, or otherwise unacceptable behavior may be)
+
+[//]: # (reported by contacting the project team at [INSERT EMAIL ADDRESS]. All)
+
+[//]: # (complaints will be reviewed and investigated and will result in a response that)
+
+[//]: # (is deemed necessary and appropriate to the circumstances. The project team is)
+
+[//]: # (obligated to maintain confidentiality with regard to the reporter of an incident.)
+
+[//]: # (Further details of specific enforcement policies may be posted separately.)
+
+[//]: # ()
+[//]: # (Project maintainers who do not follow or enforce the Code of Conduct in good)
+
+[//]: # (faith may face temporary or permanent repercussions as determined by other)
+
+[//]: # (members of the project's leadership.)
+
+### Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

LICENSE

@@ -2,7 +2,7 @@
 
 MIT License
 
-Copyright (c) 2022, Robin van de Water, Hendrik Schmidt, Patrick Rockenschaub
+Copyright (c) 2023, Robin van de Water, Hendrik Schmidt, Patrick Rockenschaub
 Copyright (c) 2021, ETH Zurich, Biomedical Informatics Group; ratschlab
 
 Permission is hereby granted, free of charge, to any person obtaining a copy

PAPER.md

@@ -0,0 +1,108 @@
+> 📋 This file follows the template for releasing ML research code
+> from [papers with code](https://github.com/paperswithcode/releasing-research-code)
+
+![YAIB](docs/figures/yaib_logo.png)
+
+# Yet Another ICU Benchmark: _A Flexible Multi-Center Framework for Clinical ML_
+
+This repository is the official implementation of [placeholder](https://arxiv.org/abs/2030.12345).
+See a graphical overview of our framework below:
+![yaib_flow](docs/figures/yaib_flow_combined.svg)
+We propose Yet Another ICU Benchmark. It was designed to address the issues of reproduciblity and provide a unified interface to develop
+clinical prediction models for the ICU. An experiment in YAIB consists of four steps: 
+1) Defining clinical concepts from the raw data.
+2) Extracting the patient cohort and specifying the prediction task.
+3) Preprocessing and feature generation.
+4) Training and evaluation of the ML model.
+
+## 📋 Requirements
+
+YAIB can be installed using conda or pip. Below you will find the three CLI commands to install YAIB using conda.
+The
+
+The first command will install an environment based on Python 3.10 (currently).
+This should work on x86 hardware.
+
+```
+conda env update -f environment.yml
+```
+
+We then activate the environment and install a package called `icu-benchmarks`, after which YAIB should be operational.
+
+```
+conda activate yaib
+pip install -e .
+```
+
+To get the datasets for this paper, please see the [YAIB-cohorts repository](https://github.com/rvandewater/YAIB-cohorts) and
+the [page on the YAIB wiki](https://github.com/rvandewater/YAIB/wiki/Generating-Cohorts). You
+will need to get access to the ICU datasets that you want to run by following a credentialing procedure.
+
+## Training
+
+The easiest method to train the models in the paper is to run these commands from the directory root:
+
+```train
+wandb sweep --verbose experiments/benchmark_classification.yml
+wandb sweep --verbose experiments/benchmark_regression.yml
+```
+
+This will create two hyperparameter sweeps for WandB for the classification and regression tasks.
+This configuration will train all the models in the paper. You can then run the following command to train the models:
+
+```train
+wandb agent <sweep_id>
+```
+
+> Tip: You can choose to run each of the configurations on a SLURM cluster instance by `wandb agent --count 1 <sweep_id>`
+
+### Quickstart
+
+If you do not yet have access to the ICU datasets, you can run the following command to train models for the included demo
+(MIMIC-III and eICU) task
+cohorts:
+
+```train
+wandb sweep --verbose experiments/demo_benchmark_classification.yml
+wandb sweep --verbose experiments/demo_benchmark_regression.yml
+```
+
+Use the command above to create a sweep and run this sweep.
+
+## Evaluation
+
+Evaluation will happen automatically after running this command. Additionally, YAIB will generate extensive log files and
+model files. The logging location is specified within the `.yml` files. We recommend using the `wandb` web-interface to inspect
+the results (see your personal WandB project.
+
+## Pre-trained Models
+
+You can download pretrained models here: [YAIB-models GitHub repository](https://github.com/rvandewater/YAIB-models).
+YAIB has built-in functionality to evaluate these models. See the below command for an example:
+
+```
+icu-benchmarks evaluate \
+    -d demo_data/mortality24/eicu_demo \
+    -n eicu_demo \
+    -t BinaryClassification \
+    -tn Mortality24 \
+    -m LGBMClassifier \
+    --generate_cache \
+    --load_cache \
+    -s 2222 \
+    -l ../yaib_logs \
+    -sn mimic \
+    --source-dir ../yaib_logs/mimic_demo/Mortality24/LGBMClassifier/2022-12-12T15-24-46/fold_0
+```
+
+## 📊Results
+
+The current latest results are shown below. Note that there have been major changes between the classification and regression
+task experiments. However, results should be comparable overall. Updated results will be posted in the near future.
+![Results](docs/figures/results_yaib.png)
+
+## Contributing
+
+This source code is released under the MIT license, included [here](LICENSE). We do not own any of the datasets used or
+included in this repository. The demo datasets have been released under
+an [Open Data Commons Open Database License (ODbL)](https://opendatacommons.org/licenses/odbl/1-0/).