Skip to content

Commit

Permalink
Updates for Docker (#947, #948)
Browse files Browse the repository at this point in the history
* Update Dockerfiles and image build script to match output
* Add full Dockerfile
* Update Docker docs
  • Loading branch information
tahorst committed Aug 21, 2020
1 parent 627a5c5 commit 457e914
Show file tree
Hide file tree
Showing 6 changed files with 77 additions and 21 deletions.
19 changes: 15 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,23 @@ You can reach us at [AllenCenterCovertLab](mailto:[email protected]

See [docs/README.md](docs/README.md) for more info on setting up and running the model.

In short, there are two alternative setups to run the model: inside a Docker container vs. in a manually constructed `pyenv` virtual environment.

In short, there are two alternative setups to run the model: inside a Docker container vs. in a manually constructed `pyenv` virtual environment. With Docker, you can start running a simulation with three commands:
1. Pull the Docker image:
```shell script
docker pull docker.pkg.github.com/covertlab/wholecellecolirelease/wcm-full:latest
```
1. Run the Docker container:
```shell script
docker run --name=wcm -it --rm docker.pkg.github.com/covertlab/wholecellecolirelease/wcm-full
```
1. Inside the container, run the model:
```shell script
python runscripts/manual/runSim.py
```

## Quick start

When running this code, prepare with these steps (the wcm-code Docker container already prepares this for you):
When running this code, prepare with these steps (the wcm-full Docker container already prepares this for you):

1. `cd` to the top level of your `wcEcoli` directory.
2. Set the `$PYTHONPATH`:
Expand Down Expand Up @@ -46,7 +57,7 @@ These scripts have command line interfaces built on Python's `argparse`, so you
**NOTE:** _Use the `-h` or `--help` switch to get complete, up-to-date documentation on the command line options._ Below are just _some_ of the command line options.


To run the parameter calculator (ParCa), which is needed to prepare input data for the simulation:
To run the parameter calculator (ParCa), which is needed to prepare input data for the simulation (this step has already been run when building the wcm-full Docker image and can be skipped if running a container from that image):
```bash
python runscripts/manual/runFitter.py [-h] [--cpus CPUS] [sim_outdir]
```
Expand Down
13 changes: 11 additions & 2 deletions cloud/build-containers.sh
Original file line number Diff line number Diff line change
@@ -1,13 +1,22 @@
#!/bin/sh
#! /usr/bin/env bash
# Build the WCM Docker container images locally.
#
# ASSUMES: The current working dir is the wcEcoli/ project root.

set -eu

# On macOS, build with `NO_AVX2=1` to avoid the OpenBLAS 0.3.6+ self-test failures
# and bad results when building with Docker Desktop on macOS.
if [ "$(uname -s)" == Darwin ]; then NO_AVX2=1; else NO_AVX2=0; fi

# Docker image #1: The Python runtime environment.
docker build -f cloud/docker/runtime/Dockerfile -t wcm-runtime .
docker build -f cloud/docker/runtime/Dockerfile -t wcm-runtime \
--build-arg NO_AVX2=$NO_AVX2 .

# Docker image #2: The Whole Cell Model code on the runtime environment.
# See this Dockerfile for usage instructions.
docker build -f cloud/docker/wholecell/Dockerfile -t wcm-code .

# Docker image #3: The Whole Cell Model code with parameters calculated, ready for sims.
# See this Dockerfile for usage instructions.
docker build -f cloud/docker/full/Dockerfile -t wcm-full .
27 changes: 27 additions & 0 deletions cloud/docker/full/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Container image #3: wcm-full.
# This Dockerfile builds a container image with the full Whole Cell Model, ready
# to run simulations, layered on the wcm-code image.
#
# To build locally from the wcEcoli/ project root directory:
#
# > docker build -f cloud/docker/full/Dockerfile -t wcm-full .
#
# After building locally you can start up a new container from the image:
#
# > docker run --name wcm -it --rm wcm-full
#
# It will start a shell where you can execute commands:
#
# # nosetests
#
# If this succeeds you should be good to go, e.g.:
#
# # python runscripts/manual/runSim.py


ARG from=wcm-code:latest
FROM ${from}

RUN python runscripts/manual/runFitter.py

CMD ["/bin/bash"]
8 changes: 4 additions & 4 deletions cloud/docker/runtime/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,21 @@
# Add option `--build-arg from=ABC` to build from a different base image but
# DO NOT USE an alpine base since the simulation math comes out different!
# See https://pythonspeed.com/articles/alpine-docker-python/ for more reasons.
ARG from=python:2.7.16
ARG from=python:2.7.16-stretch
FROM ${from}

# Option `--build-arg NO_AVX2=1` controls OpenBLAS' use of AVX2 instructions:
#
# * NO_AVX2=1 is needed to build or run properly in Docker-for-Mac due to a
# * NO_AVX2=1 is needed to build properly in Docker-for-Mac due to a
# Docker bug.
# * NO_AVX2=0 reportedly runs 20-30% faster BLAS if you're sure it'll only
# build and run in Docker-for-Linux, but this only saves ~7% in a cell sim.
# build in Docker-for-Linux, but this only saves ~7% in a cell sim.
#
# Docker-for-Mac bug:
# https://github.com/xianyi/OpenBLAS/issues/2244
# https://github.com/docker/for-mac/issues/4576
# https://github.com/machyve/xhyve/issues/171
ARG NO_AVX2=1
ARG NO_AVX2=0
ENV NO_AVX2="$NO_AVX2"

RUN apt-get update \
Expand Down
5 changes: 4 additions & 1 deletion cloud/docker/wholecell/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ ENV PYTHONPATH=/wcEcoli
RUN umask 000 && rm -rf fixtures && mkdir fixtures \
&& chmod a+rwx reconstruction/ecoli/dataclasses/process

RUN echo 'cat cloud/docker/welcome.txt' >> $HOME/.bashrc
# Show a welcome message with tips on wcEcoli.
# Copy .bashrc into / for users w/o home dirs such as `docker run --user ...`
RUN (echo 'cat cloud/docker/welcome.txt' >> ~/.bashrc \
&& cp ~/.bashrc /)

CMD ["/bin/bash"]
26 changes: 16 additions & 10 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,18 @@ These are the docs for a variety of topics on the Whole Cell Model.
There are two alternative ways to set up to run the model:

1. **Docker Container:** Install the
[Docker Desktop software](https://www.docker.com/products/docker-desktop),
build a Docker Image, then run the Docker Image as a Container.

Build the wcEcoli Docker Image this way:
[Docker Desktop software](https://www.docker.com/products/docker-desktop)
and run the Docker Image as a Container.

Pull the full docker image from the package registry in this repo:
```shell script
cd $YOUR_CODE_PROJECTS_DIR/wcEcoli # or wherever you cloned the wcEcoli project to
cloud/build-containers.sh
docker pull docker.pkg.github.com/covertlab/wholecellecolirelease/wcm-full:latest
```

You can then run the wcEcoli model inside the Container like this:

```shell script
docker run --name=wcm -it --rm wcm-code
docker run --name=wcm -it --rm docker.pkg.github.com/covertlab/wholecellecolirelease/wcm-full
```

The `-it` option starts an interactive shell.
Expand All @@ -30,10 +28,11 @@ There are two alternative ways to set up to run the model:
exit so you don't have to remember to delete old Containers.

You can mount your local directory `wcEcoli/out/` into the Container to preserve the
program's output files when the Container exits:
program's output files when the Container exits - just be sure to provide a full path
to `out/` (eg. `$PWD/out`), not just a relative path from your current directory:

```shell script
docker run --name=wcm -v $PWD/out:/wcEcoli/out -it wcm-code
docker run --name=wcm -v $PWD/out:/wcEcoli/out -it docker.pkg.github.com/covertlab/wholecellecolirelease/wcm-full
```

In this case, the output files will be owned by root. You can fix
Expand All @@ -45,7 +44,7 @@ There are two alternative ways to set up to run the model:

**NOTE:** If you encounter memory issues while using Docker Desktop (the default allocated memory is 2GB) and the simulation processes get killed midway, click the Docker icon > Preferences > Advanced > adjust memory to 4GB.

**NOTE:** Docker Desktop for Windows is not currently compatible with VirtualBox. If you use VirtualBox, try installing the legacy [Docker Toolbox](https://github.com/docker/toolbox/releases) instead. You may also need to adjust the memory allocated to the VirtualBox VM (named 'default') that gets created. In VirtualBox, select the 'default' VM and under system, change the base memory from 1 GB to 4 GB.
**NOTE:** When setting up Docker Desktop for Windows, it is best to [use the WSL2 backend](https://docs.docker.com/docker-for-windows/wsl/). If not, you might run into compatability issues when trying to run Docker, especially if you have VirtualBox installed, which could require using the legacy [Docker Toolbox](https://github.com/docker/toolbox/releases).

Inside the Container you can then run commands like these:

Expand All @@ -58,6 +57,13 @@ There are two alternative ways to set up to run the model:
**Tip:** Eventually, you'll want to delete the Docker Image. Refer to the
commands `docker image prune`, `docker image ls`, and `docker image rm`.

**Tip:** You can build your own Docker image instead of the one provided using these steps:

```shell script
cd $YOUR_CODE_PROJECTS_DIR/wcEcoli # or wherever you cloned the wcEcoli project to
cloud/build-containers.sh
```

2. **Python virtual environment:** Follow [Required development tools](dev-tools.md) to install the development tools including pyenv, gcc, make, and git, then follow [Creating the "pyenv" runtime environment](create-pyenv.md) to set up the Python runtime virtual environment for the model including binary libraries and Python packages.

You can then run wcEcoli in this Python virtualenv.
Expand Down

0 comments on commit 457e914

Please sign in to comment.