-
Notifications
You must be signed in to change notification settings - Fork 6
Container Image Versions
Container images are built and published to GHCR (GitHub Container Repository) -- see Deploying VRO. The images are tagged with an image tag, representing the version of the image. The image tag (or "image version" or "version") is:
- the first 7 of commit hash when the image is built as part of Continuous Integration (CI CD Workflows),
- or a semantic version string formatted as
v1.2.3when a release tag is created, typically for testing in LHDI environments, - or a semantic version string formatted as
release-1.2.3when a manual release tag is created, typically for deployment toprodLHDI.
See different image versions in the GitHub Container Repository. The latest image tag always refers to the last published image, which may or may not be signed by SecRel.
Previously, there was a single image tag for all container images and all containers were SecRel-scanned and deployed, regardless of whether a container had changed or not. With this constraint, as the number of images increases, more time is spent on SecRel scans, addressing SecRel alerts for unchanged images, and redeploying unchanged images. This coupling of container images to a single image version imposes an unnecessary constraint.
As described in Allow deployed containers to have different release versions #1725, we want a way to be able to deploy only the containers that have been updated, with minimal additional manual maintenance overhead. The remaining sections describe how this is achieved.
The image_vars.src and image_versions.src files determine which versions of each container is used during deployment. The image_vars.src file should not be modified manually. The versions can be overridden by manually-specified versions in GitHub Action workflows (e.g., Update deployment).
To see the latest release versions for each image, run bash scripts/image_vars.src imageVersions as described in .
- pinning an image version = set the version for a particular container image so that:
- the image is not republished to GHCR and SecRel does not rescan the (same) image
- the pinned version is used during LHDI deployments
- (The version number has the format
v*.*.*orrelease-*.*.*)
- unpinning an image version = unset the version for a particular container image so that:
- a new image is built and SecRel scans the new image
- the new image is deployed to LHDI
- (The version number is the first 7 of commit hash)
When a release is created (create-release-tag.yml), image versions that are unpinned will become pinned to that specified release version.
When the codebase is changed (e.g., a PR is merged or develop is changed) and new images will be published to GHCR, if the container image has changed for image versions that are pinned, then it will be unpinned.
Pinning and unpinning is performed by image-version.sh, which updates image_versions.src and is automatically called when certain GitHub events trigger GitHub Action workflows. The result of the script is that images that haven't changed will stay at their pinned versions, while change images will be unpinned. By looking at image_versions.src, it is apparent which images have not changed.
To manually pin images so that they don't get automatically updated, use the export myimage_VER="v1.2.3" syntax in the image_versions.src file so that they will be ignored by the image-version.sh script.
Image versions are unpinned when the image has changed. There are many reasons for an image to change:
- we updated our code used in the image (to add features or fix bugs)
- application dependencies or OS libraries have been upgraded (to address SecRel alerts)
- the base container image was updated (to get updated features or address SecRel alerts)
The image-version.sh script detects changes to images using the container-diff utility, which identifies every tiny difference. Comparison is done using the "size" and "history" analyzers. However, there are docker-build-time differences (e.g., random file identifiers, installing OS packages and libraries, and file creation timestamps) that cause two images that are practically the same to be considered different. The script's isImageSame() function tries to account for some trivial differences but it's not perfect -- e.g., any image size difference cannot be discounted and so the images must be considered different. Also in rare cases where images with identical sizes aren't necessarily the same, it also checks the "history" (of Dockerfile commands) to mitigate this risk.
(Further improvement for better image-change detection is welcome.)
Workaround: In those rare cases where a change is not detected and to ensure that a new image is published, simply unpin the image version by removing the corresponding line in the image_versions.src file.
The container-diff utility reports differences for the following images, even when there may be no apparent changes:
-
vro-app- has it's own Dockerfile; uses Java Spring -
vro-cc-app- has it's own entrypoint.sh; Python image; container-diff reports differences in pip-related files -
vro-svc-bgs-api- has it's own Dockerfile; Ruby image
Possible causes:
- OS packages are not set to specific version (search for
hadolint ignore=DL3018in Dockerfiles). As a result, new files are indeed being saved to the image. - Application libraries (e.g., pip packages) are being downloaded and cached in the image. The cache may have an updated catalog of available packages or the cache may be generating random filenames.
Further work is needed to improve the change detection mechanism.
To have SecRel scan all images, manually run SecRel with publish mode all, which will publish and then scan all images.
This is useful to identify newly discovered vulnerabilities in unchanged/pinned images.
To help ensure new vulnerabilities are addressed and to express a desire for a new version of the image, simply unpin the image version by removing the corresponding line in the image_versions.src file.
In July 2024, a bug was resolved regarding the Image Publishing functionality of the SecRel Workflow. The resolution helped to clarify our desired behavior for VRO's Image Publishing Workflow.