Updating the README and templates

README.md

@@ -3,21 +3,94 @@
 # [docker-builds](#)
 This repository contains the Dockerfiles and other assorted files necessary for building Docker images for a variety of programs used by members of the StaPH-B (State Public Health Lab Bioinformatics) consortium. The purpose of this repository is to provide a centralized location for Docker images that is easily accessible for users, with clear documentation on how the containers were built and how to use them.
 
-If you would like to contribute with a Docker image or improve upon the existing images, please fork the repository, make your changes/additions, and submit a pull request. If you are having an issue with an existing image, please submit an issue. We welcome any and all feedback!
-[See more details on how to contribute here](https://staph-b.github.io/docker-builds/contribute/)
+There are several community projects that create and share containers for bioinformatic tools. This repository contains the Dockerfiles and other assorted files necessary for building Docker images for a variety of tools used by members of the StaPH-B (State Public Health Lab Bioinformatics) consortium. The purpose of this repository is to provide a centralized location for Docker images that is easily accessible for users, with clear documentation on how the containers were built and how to use them.
 
-## [Docker User Guide](https://staph-b.github.io/docker-builds/)
-We have also created a user guide that outlines methods and best practices for using and developing docker containers.
-[Docker User Guide](https://staph-b.github.io/docker-builds/)
+This is a community resource, built and maintined by users from varied backgrounds and expertise levels. As such, we have provided some [templates for contributing to this repository](./dockerfile-template). If **you** would like to add a Docker image or improve upon the existing images, please fork the repository, make your changes/additions, and submit a pull request. If you are having an issue with an existing image, please submit an issue. We welcome any and all feedback!
 
-#### What about Singularity?
-For many people Docker is not an option, but Singularity is. Most Docker containers are compatible with Singularity and can easily be converted to Singularity format. Please see the User Guide linked above to for instructions on how to download docker images from dockerhub and how to run them using Singularity. We've worked hard to ensure that our containers are compatibile with Singularity, but if you find one that isn't, please leave an issue and let us know!
+[See more details on how to contribute here](https://staph-b.github.io/docker-builds/contribute/)
 
 ## Docker image repositories & hosting
-We host all of our docker images on two different repositories and periodically sync the images between the two:
 
-  1. Dockerhub - https://hub.docker.com/r/staphb/
-  2. Quay.io - https://quay.io/organization/staphb/
+We host all of our docker images on two different repositories:
+
+  1. [Dockerhub - https://hub.docker.com/r/staphb/](https://hub.docker.com/r/staphb/)
+  2. [Quay.io - https://quay.io/organization/staphb/](https://quay.io/organization/staphb/)
+
+The development process of creating a new image is summarized as follows:
+
+```mermaid
+graph TD
+     A[fork staphb/docker-builds repo]-->B[create tool/version directory]
+     B-->C[create readme]
+     B-->D[create dockerfile]
+     D-->G[create app and test layers]
+     A-->E[add License to Program_Licenses.md]
+     A-->F[add tool to list in this readme]
+     E-->H[submit PR]
+     F-->H
+     G-->H
+     C-->H
+     H-->I{build to test}
+     I--success-->J[PR merged and docker image pushed to dockerhub and quay]
+     I--failure-->K[edit dockerfile]
+     K-->I
+```
+
+## User Guide
+
+The [StaPH-B Docker User Guide](https://staphb.org/docker-builds/) was created to outline methods and best practices for using and developing docker containers. There are chapters for:
+
+- [Contributing](https://staphb.org/docker-builds/contribute/)
+- [Downloading Docker images](https://staphb.org/docker-builds/get_containers/)
+- [Running Docker containers](https://staphb.org/docker-builds/run_containers/)
+- [Developing Docker images](https://staphb.org/docker-builds/make_containers/)
+- [Useful links](https://staphb.org/docker-builds/useful_links/)
+
+### Summarized usage guide for docker
+
+```bash
+# Build a docker image to the 'test' layer
+docker build --tag tool:test --target test <directory to Dockerfile>
+docker build --tag samtools:test --target test samtools/1.15
+
+# Download a docker image from dockerhub (most tools have a 'latest' version tag)
+docker pull staphb/tool:version
+docker pull staphb/shigatyper:2.0.2
+
+# Run the container (don't forget to mount your volumes!)
+docker run --rm -u $(id -u):$(id -g) -v <local directory>:/data tool:version <command>
+docker run --rm -u $(id -u):$(id -g) -v $(pwd)/amrfinder_test_files:/data amrfinder:3.10 amrfinder --nucleotide 2021CK-01854_contigs.fa --threads 20 --name 2021CK-01854 --output /data/2021CK-01854.txt --organism Klebsiella
+```
+
+Further documentation can be found at [docs.docker.com](https://docs.docker.com/engine/reference/run/)
+
+### Templates
+
+Several template files are provided. These are intended to be copied and edited by contributors.
+
+1. [dockerfile-template/Dockerfile](./dockerfile-template/Dockerfile) is the basic template useful for most images
+2. [dockerfile-template/Dockerfile_mamba](./dockerfile-template/Dockerfile_mamba) is a basic template for using the micromamba base image
+3. [dockerfile-template/README.md](./dockerfile-template/README.md) is a basic readme file template to assist others in using the image
+
+### What about Singularity?
+
+For many people, Docker is not an option, but Singularity is. Most Docker containers are compatible with Singularity and can easily be converted to Singularity format. Please see the [User Guide](https://staphb.org/docker-builds/) for instructions on how to download docker images from dockerhub and how to run them using Singularity. We've worked hard to ensure that our containers are compatibile with Singularity, but if you find one that isn't, please leave an issue and let us know!
+
+### Summarized usage guide for singularity
+
+```bash
+# Pulling a container from dockerhub (creates a file)
+singularity pull --name <name of singularity file> docker://staphb/bbtools:38.96
+singularity pull --name staphb-bbtools-38.96.simg docker://staphb/bbtools:38.96
+
+# Running the container (don't forget to mount your volumes!)
+singularity exec --bind <local directory>:/data <name of singularity file> <command>
+singularity exec --bind $(pwd)/fastq:/data staphb-bbtools-38.96.simg bbduk.sh in1=sample1_R1.fastq.gz in2=sample1_R2.fastq.gz out1=bbduk/sample1_rmphix_R1.fastq.gz out2=bbduk/sample1_rmphix_R2.fastq.gz outm=bbduk/sample1.matched_phix.fq ref=/opt/bbmap/resources/phix174_ill.ref.fa.gz stats=bbduk/sample1.phix.stats.txt threads=4
+```
+
+Further documentation can be found at [docs.sylabs.io](https://docs.sylabs.io/guides/3.1/user-guide/cli.html)
+
+## Logs
 
 In November 2020, Docker began to implement pull rate limits for images hosted on dockerhub. This limits the number of `docker pull`'s per time period (e.g. anonymous users allowed 100 pulls per six hours). We applied and were approved for Docker's "Open Source Program," which should have removed the pull rate limits for all `staphb` docker images! 🎉 🥳 If you encounter an error such as `ERROR: toomanyrequests: Too Many Requests.` or `You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limits.` , please let us know by [submitting an issue.](https://github.com/StaPH-B/docker-builds/issues)
 

dockerfile-template/Dockerfile

@@ -1,60 +1,92 @@
-# FROM defines the base docker image. This command has to come first in the file
-# The 'as' keyword lets you name the folowing stage. We use `app` for the production image
-FROM ubuntu:focal as app
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+##### Thank you for using this Dockerfile template!                     #####
+##### This is an outline for the flow of building a docker image.       #####
+##### The docker image is built to the 'app' stage on dockerhub/quay.   #####
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
 
-# ARG sets environment variables during the build stage
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+##### Step 1. Set up the base image in the first stage.                 #####
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+
+# 'FROM' defines where the Dockerfile is starting to build from. This command has to come first in the file
+# The 'as' keyword lets you name the folowing stage. The production image uses everything to the 'app' stage.
+
+FROM ubuntu:jammy as app
+
+# List all software versions are ARGs near the top of the dockerfile
+# 'ARG' sets environment variables during the build stage
+# ARG variables are ONLY available during image build, they do not persist in the final image
 ARG SOFTWARENAME_VER="1.0.0"
 
-# LABEL instructions tag the image with metadata that might be important to the user
-# Optional, but highly recommended
-LABEL base.image="ubuntu:focal"
+# 'LABEL' instructions tag the image with metadata that might be important to the user
+LABEL base.image="ubuntu:jammy"
 LABEL dockerfile.version="1"
 LABEL software="SoftwareName"
-LABEL software.version=$SOFTWARENAME_VER
+LABEL software.version="${SOFTWARENAME_VER}"
 LABEL description="This software does X, Y, AND Z!"
 LABEL website="https://github.com/StaPH-B/docker-builds"
 LABEL license="https://github.com/StaPH-B/docker-builds/blob/master/LICENSE"
 LABEL maintainer="FirstName LastName"
 LABEL maintainer.email="[email protected]"
 
-# RUN executes code during the build
+# 'RUN' executes code during the build
 # Install dependencies via apt-get or yum if using a centos or fedora base
-RUN apt-get update && apt-get install -y \
+RUN apt-get update && apt-get install -y --no-install-recommends \
  dependency_a \
  dependency_b \
- dependency_c
+ dependency_c && \
+ apt-get autoclean && rm -rf /var/lib/apt/lists/*
 
 # Install and/or setup more things. Make /data for use as a working dir
-# Example: ncbi-blast+ 2.9.0
-RUN wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.9.0/ncbi-blast-2.9.0+-x64-linux.tar.gz && \
- tar -xzf ncbi-blast-2.9.0+-x64-linux.tar.gz && \
- rm ncbi-blast-2.9.0+-x64-linux.tar.gz && \
+# For readability, limit one install per 'RUN' statement.
+
+# Example: install ncbi-blast+ 2.9.0 pre-compiled linux binaries
+ARG BLAST_VER=2.9.0
+
+RUN wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/${BLAST_VER}/ncbi-blast-${BLAST_VER}+-x64-linux.tar.gz && \
+ tar -xzf ncbi-blast-${BLAST_VER}+-x64-linux.tar.gz && \
+ rm ncbi-blast-${BLAST_VER}+-x64-linux.tar.gz && \
  mkdir /data
 
-# ENV instructions set environment variables that persist from the build into the resulting image
+# 'ENV' instructions set environment variables that persist from the build into the resulting image
 # Use for e.g. $PATH and locale settings for compatibility with Singularity
-ENV PATH="/ncbi-blast-2.9.0+/bin:$PATH" \
+ENV PATH="/software-${SOFTWARENAME_VER}/bin:$PATH" \
  LC_ALL=C
 
-# WORKDIR sets working directory
+# 'CMD' instructions set a default command when the container is run. This is typically 'tool --help.'
+CMD [ "tool", "--help" ]
+
+# 'WORKDIR' sets working directory
 WORKDIR /data
 
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+##### Step 2. Set up the testing stage.                                 #####
+##### The docker image is built to the 'test' stage before merging, but #####
+##### the test stage (or any stage after 'app') will be lost.           #####
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+
 # A second FROM insruction creates a new stage
-# We use `test` for the test image
 FROM app as test
 
-# Demonstrate that the program is successfully installed
+# set working directory so that all test inputs & outputs are kept in /test
+WORKDIR /test
+
+# print help and version info; check dependencies (not all software has these options available)
+# Mostly this ensures the tool of choice is in path and is executable
+RUN softwarename --help && \
+ softwarename --check && \
+ softwarename --version
 
-# Option 1: run the program's internal tests, for example with SPAdes:
+# Demonstrate that the program is successfully installed - which is highly dependant on what the tool is.
+
+# Run the program's internal tests if available, for example with SPAdes:
 RUN spades.py --test
 
-# Option 2: write your own tests in a bash script in the same directory as your Dockerfile:
+# Option 1: write your own tests in a bash script in the same directory as your Dockerfile and copy them:
 COPY my_tests.sh .
 RUN bash my_tests.sh
 
-# Option 3: write python unit tests in a tests/ directory in the same directory as your Dockerfile:
-RUN apt-get install -y python3
-RUN mkdir tests/
-COPY tests/ tests/
-RUN python3 -m unittest discover -s tests
-
+# Option 2: write below common usage cases, for example with tb-profiler:
+RUN wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR166/009/ERR1664619/ERR1664619_1.fastq.gz && \
+    wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR166/009/ERR1664619/ERR1664619_2.fastq.gz && \
+    tb-profiler profile -1 ERR1664619_1.fastq.gz -2 ERR1664619_2.fastq.gz -t 4 -p ERR1664619 --txt

dockerfile-template/Dockerfile_builder

@@ -0,0 +1,129 @@
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+##### Thank you for using this Dockerfile template!                     #####
+##### This is an outline for the flow of building a docker image.       #####
+##### The docker image is built to the 'app' stage on dockerhub/quay.   #####
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+##### Sometimes tools required to build a tool are not needed to run    #####
+##### it. This means that images are larger than they need to be. A way #####
+##### to reduce the size of an image, is to have a stage prior to 'app' #####
+##### where these temporarily-required tools are installed. Then, only  #####
+##### relevant executables and files are copied in to the 'app' stage.  #####
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+##### Step 1. Set up the builder stage as the first stage.              #####
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+
+# Please note, all 'LABEL', 'ENV', and 'CMD' instructions will be lost
+# Also, files and executables are not carried over unless they are explicitly copied.
+
+# 'FROM' defines where the Dockerfile is starting to build from. This command has to come first in the file
+FROM ubuntu:jammy as builder
+
+# 'RUN' executes code during the build
+# Install dependencies via apt-get or yum if using a centos or fedora base
+RUN apt-get update && apt-get install -y --no-install-recommends \
+ dependency_a \
+ dependency_b \
+ dependency_c && \
+ apt-get autoclean && rm -rf /var/lib/apt/lists/*
+
+# Install and/or setup more things. Make /data for use as a working dir
+# For readability, limit one install per 'RUN' statement.
+
+# Example: install ncbi-blast+ 2.9.0 pre-compiled linux binaries
+ARG BLAST_VER=2.9.0
+
+RUN wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/${BLAST_VER}/ncbi-blast-${BLAST_VER}+-x64-linux.tar.gz && \
+ tar -xzf ncbi-blast-${BLAST_VER}+-x64-linux.tar.gz && \
+ rm ncbi-blast-${BLAST_VER}+-x64-linux.tar.gz
+
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+##### Step 2. Set up the base image in the 'app' stage.                 #####
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+
+# 'FROM' defines where the Dockerfile is starting to build from. This command has to come first in the file
+# The 'as' keyword lets you name the folowing stage. The production image uses everything to the 'app' stage.
+
+FROM ubuntu:jammy as app
+
+# List all software versions are ARGs near the top of the dockerfile
+# 'ARG' sets environment variables during the build stage
+# ARG variables are ONLY available during image build, they do not persist in the final image
+ARG SOFTWARENAME_VER="1.0.0"
+
+# 'LABEL' instructions tag the image with metadata that might be important to the user
+LABEL base.image="ubuntu:jammy"
+LABEL dockerfile.version="1"
+LABEL software="SoftwareName"
+LABEL software.version="${SOFTWARENAME_VER}"
+LABEL description="This software does X, Y, AND Z!"
+LABEL website="https://github.com/StaPH-B/docker-builds"
+LABEL license="https://github.com/StaPH-B/docker-builds/blob/master/LICENSE"
+LABEL maintainer="FirstName LastName"
+LABEL maintainer.email="[email protected]"
+
+# copy in files and executables into app stage
+COPY --from=builder <source> <destination>
+
+# example copy in blast executable
+ARG BLAST_VER=2.9.0
+COPY --from=builder /bwa/bwa-${BWA_VER}/bwa /usr/local/bin
+
+# 'RUN' executes code during the build
+# Install dependencies via apt-get or yum if using a centos or fedora base
+# Please ensure ALL dependencies for running the tool make it into this stage
+RUN apt-get update && apt-get install -y --no-install-recommends \
+ dependency_a \
+ dependency_b \
+ dependency_c && \
+ apt-get autoclean && rm -rf /var/lib/apt/lists/*
+
+# Install and/or setup more things. Make /data for use as a working dir
+# For readability, limit one install per 'RUN' statement.
+
+# 'ENV' instructions set environment variables that persist from the build into the resulting image
+# Use for e.g. $PATH and locale settings for compatibility with Singularity
+ENV PATH="/software-${SOFTWARENAME_VER}/bin:$PATH" \
+ LC_ALL=C
+
+# 'CMD' instructions set a default command when the container is run. This is typically 'tool --help.'
+CMD [ "tool", "--help" ]
+
+# 'WORKDIR' sets working directory
+WORKDIR /data
+
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+##### Step 3. Set up the testing stage.                                 #####
+##### The docker image is built to the 'test' stage before merging, but #####
+##### the test stage (or any stage after 'app') will be lost.           #####
+##### ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #####
+
+# A second FROM insruction creates a new stage
+# The test stage must be downstream from 'app'
+FROM app as test
+
+# set working directory so that all test inputs & outputs are kept in /test
+WORKDIR /test
+
+# print help and version info; check dependencies (not all software has these options available)
+# Mostly this ensures the tool of choice is in path and is executable
+RUN softwarename --help && \
+ softwarename --check && \
+ softwarename --version
+
+# Demonstrate that the program is successfully installed - which is highly dependant on what the tool is.
+
+# Run the program's internal tests if available, for example with SPAdes:
+RUN spades.py --test
+
+# Option 1: write your own tests in a bash script in the same directory as your Dockerfile and copy them:
+COPY my_tests.sh .
+RUN bash my_tests.sh
+
+# Option 2: write below common usage cases, for example with tb-profiler:
+RUN wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR166/009/ERR1664619/ERR1664619_1.fastq.gz && \
+    wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR166/009/ERR1664619/ERR1664619_2.fastq.gz && \
+    tb-profiler profile -1 ERR1664619_1.fastq.gz -2 ERR1664619_2.fastq.gz -t 4 -p ERR1664619 --txt