Skip to content

Container Image Versions

Jump to bottom
Yoom Lam edited this page Jul 8, 2023 · 16 revisions

Container images are built and published to GHCR (GitHub Container Repository) -- see Deploying VRO. The images are tagged with an image tag, representing the version of the image. The image tag (or "image version" or "version") is:

  • the first 7 of commit hash when the image is built as part of Continuous Integration (CI CD Workflows),
  • or a semantic version string formatted as v1.2.3 when a release tag is created, typically for testing in LHDI environments,
  • or a semantic version string formatted as release-1.2.3 when a manual release tag is created, typically for deployment to prod LHDI.

See different image versions in the GitHub Container Repository. The latest image tag always refers to the last published image, which may or may not be signed by SecRel.

The image_vars.src and image_versions.src files determine which versions of each container is used during deployment. The image_vars.src file should not be modified manually. The versions can be overridden by manually-specified versions in GitHub Action workflows (e.g., Update deployment).

Relevant ticket: Allow deployed containers to have different release versions #1725

Terminology

  • pinning an image version = set the version for a particular container image so that:
    • the image is not republished to GHCR and SecRel does not rescan the (same) image
    • the pinned version is used during LHDI deployments
    • (The version number has the format v*.*.* or release-*.*.*)
  • unpinning an image version = unset the version for a particular container image so that:
    • a new image is built and SecRel scans the new image
    • the new image is deployed to LHDI
    • (The version number is the first 7 of commit hash)

Pinning versions

When a release is created (create-release-tag.yml), image versions that are unpinned will become pinned to that specified release version.

Unpinning versions

When the codebase is changed (e.g., a PR is merged or develop is changed) and new images will be published to GHCR, if the container image has changed for image versions that are pinned, then it will be unpinned.

image-version.sh automation script

Pinning and unpinning is performed by image-version.sh, which updates image_versions.src and is automatically called when certain GitHub events trigger GitHub Action workflows. The result of the script is that images that haven't changed will stay at their pinned versions, while change images will be unpinned. By looking at image_versions.src, it is apparent which images have not changed.

Manually pin image versions

To manually pin images so that they don't get automatically updated, use the : ${myimage_VER:="v1.2.3"} syntax in the image_versions.src file so that they will be ignored by the image-version.sh script.

Detecting changes to images

Image versions are unpinned when the image has changed. There are many reasons for an image to change:

  • we updated our code used in the image (to add features or fix bugs)
  • application dependencies or OS libraries have been upgraded (to address SecRel alerts)
  • the base container image was updated (to get updated features or address SecRel alerts)

The image-version.sh script detects changes to images using the container-diff utility, which identifies every tiny difference. Comparison is done using the "size" and "history" analyzers. However, there are docker-build-time differences (e.g., random file identifiers, installing OS packages, and file creation timestamps) that cause two images that are practically the same to be considered different. The script's isImageSame() function tries to account for some trivial differences but it's not perfect -- e.g., any image size difference cannot be discounted and so the images must be considered different. Also in rare cases where images with identical sizes aren't necessarily the same, it also checks the "history" (of Dockerfile commands) to mitigate this risk. (Further improvement for better image-change detection is welcome.)

Workaround: In those rare cases where a change is not detected and to ensure that a new image is published, simply unpin the image version by removing the corresponding line in the image_versions.src file.

Clone this wiki locally