Dask docker images

Image	Description	Versions
daskdev/dask	Base image to use for Dask scheduler and workers
daskdev/dask-notebook	Jupyter Notebook image to use as helper entrypoint

Example

An example docker-compose.yml file is included for starting a small cluster.

docker-compose up

Open the notebook using the URL that is printed by the output so it has the token.

On a new notebook run:

from dask.distributed import Client
client = Client()  # The address is automatically set by the DASK_SCHEDULER_ADDRESS environment variable
client.ncores()

It should output something like this:

{'tcp://172.23.0.4:41269': 4}

Environment Variables

The following environment variables are supported for both the base and notebook images:

• $EXTRA_APT_PACKAGES - Space separated list of additional system packages to install with apt.
• $EXTRA_CONDA_PACKAGES - Space separated list of additional packages to install with conda.
• $EXTRA_PIP_PACKAGES - Space separated list of additional python packages to install with pip.

The notebook image supports the following additional environment variables:

• $JUPYTERLAB_ARGS - Extra arguments to pass to the jupyter lab command.

Building images

Docker compose provides an easy way to building all the images with the right context

cd build

docker-compose build

# Just build one image e.g. notebook
docker-compose build notebook

Cross building

The images can be cross-built using docker buildx bake. However buildx bake does not listen to depends_on (since in theory that is only a runtime not a build time constraint docker/buildx#447). To work around this we first need to build the "base-notebook" image.

cd build

# If you have permission to push to daskdev/
docker buildx bake --progress=plain --set *.platform=linux/arm64,linux/amd64 --push base-notebook
docker buildx bake --progress=plain --set *.platform=linux/arm64,linux/amd64 --push

# If you don'tset DOCKERUSER to your dockerhub username.
export DOCKERUSER=holdenk
docker buildx bake --progress=plain --set *.platform=linux/arm64,linux/amd64 --set base-notebook.tags.image=${DOCKERUSER}/base-notebook:lab-py38 --push base-notebook
docker buildx bake --progress=plain --set *.platform=linux/arm64,linux/amd64 --set scheduler.tags=${DOCKERUSER}/dask --set worker.tags=${DOCKERUSER}/dask --set notebook.tags=${DOCKERUSER}/dask-notebook --set base-notebook.tags=${DOCKERUSER}/base-notebook:lab-py38 --set notebook.args.base=${DOCKERUSER} --push

Releasing

Building and releasing new image versions is done automatically.

• When a new Dask version is released the watch-conda-forge action will trigger and open a PR to update the latest release version in this repo.
• If images build successfully that PR will be automatically merged by the automerge action.
• When a PR like this is merged which updates the pinned release version a tag is automatically created to match that version by the autotag action.
• When tags are created a new image is built and pushed using the docker/build-push-action action.