Release pipelines should pin their image versions

[AlanGreene]

All nightly releases started failing on September 19th due to an issue with the koparse image, see fix in #2196

The nightly releases picked up the bad image as they were using the latest tag.

All images should be pinned using an appropriate image digest to ensure repeatable builds and to avoid picking up unexpected changes. From a quick review of the first few nightly releases many projects are using the latest tag of at least the ko, ko-gcloud, and koparse images.

List of releases in the tekton-nightly namespace:

• add-pr-body runs, definition Pin images used in the add-pr-body interceptor release pipeline #2203
• add-pr-body-ci runs, definition Pin images used in the add-pr-body interceptor release pipeline #2203
• add-team-members runs, definition Pin images used in the add-team-members interceptor release pipeline #2204
• cel runs, definition Pin images used for the cel release pipeline experimental#977
• chains runs, definition Pin images used in the release pipeline chains#1215
• cloudevents runs, definition Pin images used in the cloudevents release pipeline experimental#978
• concurrency runs, definition Pin images used for the common release pipeline experimental#973
• dashboard runs, definition Pin images used for the release pipeline dashboard#3622
• operator runs, definition Pin images used in the release pipeline operator#2350
• pipeline-in-pod runs, definition Pin images used for the common release pipeline experimental#973
• pipeline-to-taskrun runs, definition Pin images used in the pipeline-to-taskrun release pipeline experimental#976
• pipelines runs, definition Pin alpine image used in release pipeline pipeline#8287
• pipelines-in-pipelines runs, definition Pin images used in the pipeline-in-pipelines release pipeline experimental#975
• pr-commenter runs, definition Pin images used in the custom tasks release pipelines #2205
• pr-status-updater runs, definition Pin images used in the custom tasks release pipelines #2205
• task-loops runs, definition Pin images used for the task-loops release pipeline experimental#974
• triggers runs, definition Migrate nightly builds and releases to ghcr.io triggers#1767
• wait-task runs, definition Pin images used for the wait-task release pipeline experimental#972
• workflows runs, definition Pin images used for the common release pipeline experimental#973

Taking the most recent one as an example, these are the unpinned images used, most are using latest:

wait-task

• nightly release PipelineRuns
• release pipeline definition

task	step	image
git-clone	clone	gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.18.1 (should specify digest too)
unit-tests	unit-test	docker.io/library/golang:latest
build-tests	build	docker.io/library/golang:latest
publish-images	create-ko-yaml	busybox
publish-images	container-registry-auth	gcr.io/go-containerregistry/crane:debug
publish-images	run-ko	gcr.io/tekton-releases/dogfooding/ko:latest
publish-images	koparse	gcr.io/tekton-releases/dogfooding/koparse:latest
publish-images	tag-images	gcr.io/go-containerregistry/crane:debug
report-bucket	create-results	alpine

[AlanGreene]

Dashboard and Pipelines nightly releases are passing again, updated to use the latest koparse image, and now publishing their images to ghcr.io.

Andrea also has a PR open against Triggers to update it to the latest koparse and move to ghcr.io.

PRs open for other projects, custom tasks, interceptors, etc. to pin images to known working state.

[AlanGreene]

Triggers nightly release is passing again, also updated to the latest koparse image and publishing images to ghcr.io

The following nightly releases are also passing again:

• Operator
• cloudevents
• concurrency
• pipeline-in-pod
• pipeline-to-taskrun
• pipelines-in-pipelines
• task-loops
• wait-task
• workflows

Chains nightly is failing due to a known bad test: tektoncd/chains#1178

cel nightly is failing due to wrong go version:

2024-09-27T02:08:39.542964111Z Error: error processing import paths in "/workspace/go/src/github.com/tektoncd/experimental/cel/config/500-controller.yaml": error resolving image references: build: go build: exit status 1: # [golang.org/x/sys/unix](http://golang.org/x/sys/unix)
2024-09-27T02:08:39.543058620Z vendor/golang.org/x/sys/unix/syscall.go:83:16: unsafe.Slice requires go1.17 or later (-lang was set to go1.16; check go.mod)

add-team-members, add-pr-body, and add-pr-body-ci nightly are failing due to missing task:

Pipeline tekton-nightly/add-team-members-release can't be Run; it contains Tasks that don't exist: Couldn't retrieve Task "": error requesting remote resource: error getting "bundleresolver" "tekton-nightly/bundles-c741b1a1ad5c6d7d07337a952fbd7d6a": could not find object in image with kind: task and name: git-batch-merge

pr-commenter and pr-status-updater are failing due to:

2024-09-27T00:01:06.270600691Z go: updates to go.mod needed; to update it:
2024-09-27T00:01:06.270635326Z 	go mod tidy

[afrittoli]

add-team-members-release

This is now working fine, I fixed the references to the wrong tasks.

add-pr-body-ci

This now failing for the same go mod tidy issue as the pr-* ones

add-pr-body

This now fails with a koparse issue

PR to fix: #2225

[afrittoli]

Fix for the CEL component: tektoncd/experimental#979

[afrittoli]

Remaining issues:

• pr-commenter:

+ IMAGES=' http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/custom-tasks/pr-commenter/cmd/pr-commenter:v20241003-d627097ada'
/tekton/scripts/script-3-bmfdd: line 19: syntax error: unterminated quoted string

• add-pr-body:

+ IMAGES_PATH=http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing
+ '[' tekton/ci/interceptors/add-pr-body '!='  ]
+ IMAGES_PATH=http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/interceptors/add-pr-body
+ IMAGES='http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/interceptors/add-pr-body/cmd/add-pr-body:v20241003-d627097ada'
+ koparse --path /workspace/output/v20241003-d627097ada/release.yaml --base http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/interceptors/add-pr-body)--images http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/interceptors/add-pr-body/cmd/add-pr-body:v20241003-d627097ada
Expected images did not match: Images ['http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/interceptors/add-pr-body/cmd/add-pr-body:v20241003-d627097ada'] were expected but missing.

• chains: still one unit test failing, the new nightly will skip unit tests so it should work once Switch chains nightly release to ghcr.io #2226 is merged

[afrittoli]

All remaining issues have been fixed.

[AlanGreene]

Thanks for seeing this over the line @afrittoli 🙇 I'll be back next week