Discussion: Image Management & Staging -> Prod workflow

[scottyhq]

This is an issue for discussing the current staging -> prod workflow and management and testing of Docker images.

The original design of hubploy works great for a single image per hub, defined in a subfolder. This is currently the configuration for the GCP hub:
https://github.com/pangeo-data/pangeo-cloud-federation/tree/staging/deployments/gcp-uscentral1b/image/binder

Over time on the aws-uswest2.pangeo.io we've found it helpful to have a handful of different images available for selection. In particular, pangeo/ml-notebook which has specific GPU config

pangeo-cloud-federation/deployments/icesat2/config/common.yaml

Lines 31 to 39 in 7e842bf

	- display_name: "Latest ML-notebook"
	description: "https://github.com/pangeo-data/pangeo-docker-images/tree/master/ml-notebook"
	kubespawner_override:
	image: pangeo/ml-notebook:latest
	mem_limit: 60G
	mem_guarantee: 25G
	environment: {'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility'}
	tolerations: [{'key': 'nvidia.com/gpu','operator': 'Equal','value': 'present','effect': 'NoSchedule'}]
	extra_resource_limits: {"nvidia.com/gpu": "1"}

Or images for special events like hackweeks:

pangeo-cloud-federation/deployments/icesat2/config/common.yaml

Lines 19 to 22 in 7e842bf

	- display_name: "ICESat-2 Hackweek 2020"
	description: "https://github.com/ICESAT-2HackWeek/jupyter-image-2020"
	kubespawner_override:
	image: uwhackweeks/icesat2:latest

A while back I started using the latest tag in the profile list because I couldn't be bothered updating tags frequently. This works pretty well if the 'latest' tag is always guaranteed as a tested production release. Which is the goal with our pangeo-docker-images repository design (https://github.com/pangeo-data/pangeo-docker-images#image-tagging-and-continuous-building). This key to using latest tags in config is to ensure that the pod always re-pulls the image (default is 'IfNotPresent') when launched:

    singleuser:
      image:
         pullPolicy: 'Always'

This still isn't ideal from a reproducibility standpoint (probably best to use explicit tags).

Possible next steps:

1. Tackle this open issue on hubploy for multiple images support Support multiple images  berkeley-dsep-infra/hubploy#10 ?

2. Add GitHub Action to watch for new tags on DockerHub and issue PRs to bump all image tags under profile list?

[salvis2]

This would be really nice. We could probably quash the issue we had a bit ago with berkeley-dsep-infra/hubploy#73 and have some listed images from DockerHub, some from ECR, etc.