generated from pythonhealthdatascience/stars_reproduction_template
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs(readme): add mostly-completed reproduce readme
- Loading branch information
1 parent
4cec7c8
commit f2e0900
Showing
1 changed file
with
112 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,44 +1,138 @@ | ||
# Reproduction README | ||
|
||
<!-- TODO: Remove this warning once filled out README --> | ||
**Please note: This is a template README and has not yet been completed** | ||
|
||
<!-- TODO: Fill out the README --> | ||
## Model summary | ||
|
||
TBC | ||
This study models Points-of-Dispensing (PODs) in New York City. These are sites set up during a public health emergency to dispense countermeasures. The authors use evolutionary algorithms combined with discrete-event simulation to explore optimal staff numbers with regards to resource use, wait time and throughput. | ||
|
||
## Scope of the reproduction | ||
|
||
TBC | ||
In this assessment, we attempted to reproduce 8 items: 6 figures and 2 tables. | ||
|
||
## Reproducing these results | ||
|
||
### Repository overview | ||
|
||
TBC | ||
```{bash} | ||
├── docker | ||
│ └── ... | ||
├── inputs | ||
│ └── ... | ||
├── python_outputs | ||
│ └── ... | ||
├── python_scripts | ||
│ └── ... | ||
├── r_outputs | ||
│ └── ... | ||
├── r_scripts | ||
│ └── ... | ||
├── renv | ||
│ └── ... | ||
├── tests | ||
│ └── ... | ||
├── .Rprofile | ||
├── environment.yml | ||
├── README.md | ||
├── renv.lock | ||
└── reproduction.Rproj | ||
``` | ||
|
||
* `docker/` - Instructions for creation of Docker container. | ||
* `inputs/` - Nine `.txt` files with parameters for pre-screening | ||
* `python_outputs/` - Results for each experiment | ||
* `python_scripts/` - Python code to run experiments with evolutionary algorithms and discrete-event simulation | ||
* `r_outputs/` - Tables and figures | ||
* `r_scripts/` - R code to produce the tables and figures from the model outputs | ||
* `renv/` - Instructions for creation of R environment | ||
* `tests/` - Test to check that model produces consistent results with our reproduction. | ||
* `.Rprofile` - Activates R environment. | ||
* `environment.yml` - Instructions for creation of python environment. | ||
* `README.md` - This file! | ||
* `renv.lock` - Lists R version and all packages in the R environment. | ||
* `reproduction.Rproj` - Project settings. | ||
|
||
### Step 1. Set up Python environment | ||
|
||
#### Option A: Conda/mamba environment | ||
|
||
A `conda`/`mamba` environment has been provided. To create this environment on your machine, you should run this command in your terminal: | ||
|
||
``` | ||
conda env create -f environment.yml | ||
``` | ||
|
||
You can then use this environment in your preferred IDE, such as VSCode (where you will be asked to select the kernel/interpreter). You can activate it in the terminal by running: | ||
|
||
``` | ||
conda activate hernandez2015 | ||
``` | ||
|
||
You can run either of these commands also using `mamba` instead (e.g. `mamba activate hernandez2015`). | ||
|
||
#### Option B: Build local docker image | ||
|
||
<!--TODO: Add docker instructions --> | ||
|
||
### Step 1. Set up environment | ||
### Step 2. Running the model | ||
|
||
TBC | ||
#### Option A: Run the `.py` files | ||
|
||
### Step 2. Running the model | ||
To run all the model scenarios, open and execute each of the `Experiment...` files in `python_scripts/`. These are: | ||
|
||
* Experiment 1 - Figure 5 and 6 and Appendix A.2 | ||
* Experiment 2 - Figure 7 | ||
* Experiment 3 - Figure 8 | ||
* Experiment 4 - Figure 9 | ||
* Experiment 5 - Figure 10 | ||
* Experiment A.1 - Table 3 | ||
|
||
Ensure you are located in the `python_scripts/` directory when running these. | ||
|
||
#### Option B: Pytest | ||
|
||
One of the experiments (Experiment 5) has been set up as a test, so you can check whether your output matches the expected output. You can run this from the terminal. Ensure that the `hernandez2015` environment is active and that you are in the `tests/` folder. Then run: | ||
|
||
``` | ||
pytest -W ignore::DeprecationWarning | ||
``` | ||
|
||
We use `ignore::DeprecationWarning` to prevent a warning that the "oldnumeric module will be dropped in Numpy 1.9". The test should take approximately 10 to 20 seconds to run. | ||
|
||
TBC | ||
### Step 3. Set up the R environment | ||
|
||
An `renv` environment has been provided. To create this environment locally on your machine, you should open the R project with the `renv` package loaded, and then run: | ||
|
||
``` | ||
renv::restore() | ||
``` | ||
|
||
In `renv.lock`, you will see the version of R listed. However, renv will not install this for you, so you will need to switch to this yourself if you wish to also use the same version of R. This reproduction has been run in R 4.4.1, and it is possible (although not definite) that later versions of R may not be compatible, or that you may encounter difficulties installing the specified package versions in later versions of R. | ||
|
||
### Step 4. Produce the figures and tables | ||
|
||
Run `r_scripts/create_figures.Rmd` to produce the tables and figures. | ||
|
||
## Reproduction specs and runtime | ||
|
||
TBC | ||
This reproduced was conducted mostly on an Intel Core i9-13900K with 81GB RAM running Pop!_OS 22.04 Linux. | ||
|
||
To run all the experiments, total run time is **9 hours 16 minutes** (556 minutes). | ||
|
||
* Experiment 1 - 4 hours 28 minutes | ||
* Experiment 2 - 58 minutes | ||
* Experiment 3 - 3 hours 13 minutes | ||
* Experiment 4 - 37 minutes | ||
* Experiment 5 - 8 seconds | ||
* Experiment A.1 - 10 seconds | ||
|
||
However, it is important to note that: | ||
|
||
<!-- Hint: On Linux, you can find specs by running the following commands in the terminal: | ||
cat /etc/os-release # Shows operating system and version | ||
lscpu # Shows computer model | ||
free -g # Shows RAM in GB (combine total column) --> | ||
* This involve running the models within each experiment in parallel, but each of the experiment files seperately. If these are run at the same time (which I could do without issue), then you will be able to run them all within **4 hours 28 minutes** (the longest experiment) | ||
* This has excluded one of the variants for Experiment 3, which I did not run as it had a very long run time (quoted to be 27 hours in the article) and as, regardless, I had not managed to reproduce the other sub-plots in the figure for that experiment | ||
|
||
## Citation | ||
|
||
TBC | ||
To cite the original study, please refer to the reference above. To cite this reproduction, please refer to the `CITATION.cff` file in the parent folder. | ||
|
||
## License | ||
|
||
TBC | ||
This repository is licensed under the MIT License. |