Skip to content

Commit

Permalink
docs(readme): add mostly-completed reproduce readme
Browse files Browse the repository at this point in the history
  • Loading branch information
amyheather committed Oct 9, 2024
1 parent 4cec7c8 commit f2e0900
Showing 1 changed file with 112 additions and 18 deletions.
130 changes: 112 additions & 18 deletions reproduction/README.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,138 @@
# Reproduction README

<!-- TODO: Remove this warning once filled out README -->
**Please note: This is a template README and has not yet been completed**

<!-- TODO: Fill out the README -->
## Model summary

TBC
This study models Points-of-Dispensing (PODs) in New York City. These are sites set up during a public health emergency to dispense countermeasures. The authors use evolutionary algorithms combined with discrete-event simulation to explore optimal staff numbers with regards to resource use, wait time and throughput.

## Scope of the reproduction

TBC
In this assessment, we attempted to reproduce 8 items: 6 figures and 2 tables.

## Reproducing these results

### Repository overview

TBC
```{bash}
├── docker
│ └── ...
├── inputs
│ └── ...
├── python_outputs
│ └── ...
├── python_scripts
│ └── ...
├── r_outputs
│ └── ...
├── r_scripts
│ └── ...
├── renv
│ └── ...
├── tests
│ └── ...
├── .Rprofile
├── environment.yml
├── README.md
├── renv.lock
└── reproduction.Rproj
```

* `docker/` - Instructions for creation of Docker container.
* `inputs/` - Nine `.txt` files with parameters for pre-screening
* `python_outputs/` - Results for each experiment
* `python_scripts/` - Python code to run experiments with evolutionary algorithms and discrete-event simulation
* `r_outputs/` - Tables and figures
* `r_scripts/` - R code to produce the tables and figures from the model outputs
* `renv/` - Instructions for creation of R environment
* `tests/` - Test to check that model produces consistent results with our reproduction.
* `.Rprofile` - Activates R environment.
* `environment.yml` - Instructions for creation of python environment.
* `README.md` - This file!
* `renv.lock` - Lists R version and all packages in the R environment.
* `reproduction.Rproj` - Project settings.

### Step 1. Set up Python environment

#### Option A: Conda/mamba environment

A `conda`/`mamba` environment has been provided. To create this environment on your machine, you should run this command in your terminal:

```
conda env create -f environment.yml
```

You can then use this environment in your preferred IDE, such as VSCode (where you will be asked to select the kernel/interpreter). You can activate it in the terminal by running:

```
conda activate hernandez2015
```

You can run either of these commands also using `mamba` instead (e.g. `mamba activate hernandez2015`).

#### Option B: Build local docker image

<!--TODO: Add docker instructions -->

### Step 1. Set up environment
### Step 2. Running the model

TBC
#### Option A: Run the `.py` files

### Step 2. Running the model
To run all the model scenarios, open and execute each of the `Experiment...` files in `python_scripts/`. These are:

* Experiment 1 - Figure 5 and 6 and Appendix A.2
* Experiment 2 - Figure 7
* Experiment 3 - Figure 8
* Experiment 4 - Figure 9
* Experiment 5 - Figure 10
* Experiment A.1 - Table 3

Ensure you are located in the `python_scripts/` directory when running these.

#### Option B: Pytest

One of the experiments (Experiment 5) has been set up as a test, so you can check whether your output matches the expected output. You can run this from the terminal. Ensure that the `hernandez2015` environment is active and that you are in the `tests/` folder. Then run:

```
pytest -W ignore::DeprecationWarning
```

We use `ignore::DeprecationWarning` to prevent a warning that the "oldnumeric module will be dropped in Numpy 1.9". The test should take approximately 10 to 20 seconds to run.

TBC
### Step 3. Set up the R environment

An `renv` environment has been provided. To create this environment locally on your machine, you should open the R project with the `renv` package loaded, and then run:

```
renv::restore()
```

In `renv.lock`, you will see the version of R listed. However, renv will not install this for you, so you will need to switch to this yourself if you wish to also use the same version of R. This reproduction has been run in R 4.4.1, and it is possible (although not definite) that later versions of R may not be compatible, or that you may encounter difficulties installing the specified package versions in later versions of R.

### Step 4. Produce the figures and tables

Run `r_scripts/create_figures.Rmd` to produce the tables and figures.

## Reproduction specs and runtime

TBC
This reproduced was conducted mostly on an Intel Core i9-13900K with 81GB RAM running Pop!_OS 22.04 Linux.

To run all the experiments, total run time is **9 hours 16 minutes** (556 minutes).

* Experiment 1 - 4 hours 28 minutes
* Experiment 2 - 58 minutes
* Experiment 3 - 3 hours 13 minutes
* Experiment 4 - 37 minutes
* Experiment 5 - 8 seconds
* Experiment A.1 - 10 seconds

However, it is important to note that:

<!-- Hint: On Linux, you can find specs by running the following commands in the terminal:
cat /etc/os-release # Shows operating system and version
lscpu # Shows computer model
free -g # Shows RAM in GB (combine total column) -->
* This involve running the models within each experiment in parallel, but each of the experiment files seperately. If these are run at the same time (which I could do without issue), then you will be able to run them all within **4 hours 28 minutes** (the longest experiment)
* This has excluded one of the variants for Experiment 3, which I did not run as it had a very long run time (quoted to be 27 hours in the article) and as, regardless, I had not managed to reproduce the other sub-plots in the figure for that experiment

## Citation

TBC
To cite the original study, please refer to the reference above. To cite this reproduction, please refer to the `CITATION.cff` file in the parent folder.

## License

TBC
This repository is licensed under the MIT License.

0 comments on commit f2e0900

Please sign in to comment.