CI: use local registry, part 1 of 2

As of containers/automation_images#357
our CI VMs include a local registry preloaded with all(*)
images used in tests.

 * where "all" means "most".

This commit defines a new registries-cached.conf, used in tests,
that redirects docker and quay to the new local registry. The
hope is that this will reduce CI flakes.

Since tests change over time, and new tests may require new
images, this commit also adds a mechanism for pulling in
remote images at test run time. Obviously this negates
the purpose of the cache, since it introduces a flake
pain point. The idea is: DO NOT DO THIS UNLESS ABSOLUTELY
NECESSARY, and then, if we have to do this, hurry up and
spin new CI VMs that include the new image(s).

Signed-off-by: Ed Santiago <[email protected]>

.cirrus.yml

@@ -32,7 +32,7 @@ env:
     DEBIAN_NAME: "debian-13"
 
     # Image identifiers
-    IMAGE_SUFFIX: "c20240620t153000z-f40f39d13"
+    IMAGE_SUFFIX: "c20240708t152000z-f40f39d13"
     FEDORA_CACHE_IMAGE_NAME: "fedora-${IMAGE_SUFFIX}"
     PRIOR_FEDORA_CACHE_IMAGE_NAME: "prior-fedora-${IMAGE_SUFFIX}"
     DEBIAN_CACHE_IMAGE_NAME: "debian-${IMAGE_SUFFIX}"

contrib/cirrus/lib.sh

@@ -68,6 +68,10 @@ CIRRUS_BASE_SHA=${CIRRUS_BASE_SHA:-unknown$(date +%d)}  # difficult to reliably
 CIRRUS_BUILD_ID=${CIRRUS_BUILD_ID:-unknown$(date +%s)}  # must be short and unique enough
 CIRRUS_TASK_ID=${CIRRUS_BUILD_ID:-unknown$(date +%d)}   # to prevent state thrashing when
                                                         # debugging with `hack/get_ci_vm.sh`
+
+# All CI jobs use a local registry
+export CI_USE_REGISTRY_CACHE=true
+
 # Regex defining all CI-related env. vars. necessary for all possible
 # testing operations on all platforms and versions.  This is necessary
 # to avoid needlessly passing through global/system values across

contrib/cirrus/setup.sh

@@ -87,6 +87,18 @@ echo "Configuring /etc/containers/registries.conf"
 mkdir -p /etc/containers
 echo -e "[registries.search]\nregistries = ['docker.io', 'registry.fedoraproject.org', 'quay.io']" | tee /etc/containers/registries.conf
 
+# As of July 2024, CI VMs come built-in with a registry.
+LCR=/var/cache/local-registry/local-cache-registry
+if [[ -x $LCR ]]; then
+    # Images in cache registry are prepopulated at the time
+    # VMs are built. If any PR adds a dependency on new images,
+    # those must be fetched now, at VM start time. This should
+    # be rare, and must be fixed in next automation_images build.
+    while read new_image; do
+        $LCR cache $new_image
+    done < <(grep '^[^#]' tests/NEW-IMAGES || true)
+fi
+
 show_env_vars
 
 if [[ -z "$CONTAINER" ]]; then

tests/NEW-IMAGES

@@ -0,0 +1,15 @@
+#
+# As of July 2024, all CI VMs include a local registry prepopulated
+# with all container images used in tests:
+#
+#     https://github.com/containers/automation_images/pull/357
+#     https://github.com/containers/podman/pull/22726
+#     https://github.com/containers/buildah/pull/5584
+#
+# From time to time -- infrequently, we hope! -- existing images are
+# updated, or tests are added that require new images. Those must be
+# prefetched on each CI job, at least until new VMs are built. This
+# file contains those images.
+#
+# Format is one FQIN per line. Enumerate them below:
+#

tests/registries-cached.conf

@@ -0,0 +1,19 @@
+# Note that changing the order here may break tests.
+unqualified-search-registries = ['docker.io', 'quay.io', 'registry.fedoraproject.org']
+
+[[registry]]
+# As of July 2024, all CI VMs come with a local registry prepopulated
+# with all images needed by tests.
+prefix="docker.io"
+location="127.0.0.1:60333"
+insecure=true
+
+[[registry]]
+prefix="quay.io"
+location="127.0.0.1:60333"
+insecure=true
+
+[[registry]]
+prefix="docker.io/library"
+location="127.0.0.1:60333/libpod"
+insecure=true