TSSC Multi-CI Development for RHTAP

This repository contains the innerloop development environment for the Multi-CI translations from the RHTAP Pipelines. This repository can also be directly tested in the CI system as it includes a copy of a sample source app for RHTAP (node.js) as well as a gitops repo for that deployment. This may be moved to a separate repo in future dev mode builds.

This repo includes Jenkins, Gitlab and Github Actions tests of the scripts

The tasks appear in the rhtap directory and are updated manually. Once updated they can be tested locally in the shell or pushed to the appropriate CI system and tested in that CI.

This includes:

1. A Jenkins shared library to be tested in Jenkins as well as Developer Hub as part of RHTAP.
2. A gitlab CI file which can be pushed to gitlab in a fork of this repo to test it
3. A github actions workflow which can be run manually in this repo or pushed to a test repo to validate. When github actions are created for some of the tasks, this will require publishing to github individual to test.

Development Mode

In development mode, the pipeline scripts can be tested using local shell scripts.

bash build-pipeline.sh to run a build which will create the Image, SBOM and other artifacts from your local shell. bash promote-pipeline to run a promotion script which will run the promotion flow, upload SBOM (some systems) and validate Enterprise Contract.

The local execution requires binaries to be installed in your cluster on your path. The shell scripts will print error message if any binaries are missing.

RHTAP-integrated mode

To test the pipelines in a real RHTAP environment:

1. Release the Templates to your forks
2. Deploy RHTAP after changing the DEVELOPER_HUB__CATALOG__URL in your .env (see .env.template) to point to your fork
3. Go to the Developer Hub url in your RHTAP instance, sign in and create a new component
   • Select the CI type that you want to test (Tekton, Jenkins, GitHub actions, ...)
4. Trigger the pipeline that you want to test (typically by creating or merging a PR)
   • Jenkins pipelines need some manual setup first, see Configuring Jenkins and Setting up Jenkins jobs

Deploying RHTAP for development

You'll need an OpenShift cluster for deploying RHTAP. Find the supported OpenShift versions in the Release notes for the latest version of RHTAP (in the Compatibility and support matrix section). If you have acess to the Red Hat Internal Slack workspace, a good way to provision a cluster can be to message Cluster Bot: rosa create <version> 6h.

Once you have a cluster ready:

1. Set up your .env file (or .envrc or whatever you prefer). Copy the .env.template file and fill in the values according to the inline instructions.
2. cd to your local clone of the https://github.com/redhat-appstudio/rhtap-cli repo
   • Compile the CLI with make if you haven't done so yet
3. Source your .env file
4. Run integration-tests/scripts/install.sh
5. When finished, the script will print a Homepage URL, Webhook URL and Callback URL. Go to the GitHub app that you used for RHTAP integration and set these urls in the settings.

Note: once you've set up your .env for the first time, most of the variables will be re-usable for future deployments.

Releasing to Templates and Jenkins Library

Jenkins

In order to run in RHTAP via software templates, you need to release to a fork of templates https://github.com/redhat-appstudio/tssc-sample-templates and install these into RHTAP. When validated in a fork, send a pull request to the release templates repo.

In order to run the Jenkinsfile you must push to the Jenkins library https://github.com/redhat-appstudio/tssc-sample-jenkins If you want to use a fork you must update your jenkinsfile to reference your fork repository in the jenkins file.

library identifier: 'RHTAP_Jenkins@main', retriever: modernSCM(
  [$class: 'GitSCMSource',
   remote: 'https://github.com/redhat-appstudio/tssc-sample-jenkins.git'])

To update forks, in preparation for sending pull requests to the official library locations, you can run bash hack/copy-to-tssc-templates to update your local forked repos and then manually check and push to your branch.

TODO - update to include gitlab and gihub

Creating a cosign signing secret

Take a look in hack/create-sign-secret. If you run the build pipeline in development mode this script would have already been used to create the git ignored file rhtap/signing-secret-env.sh, but you can run it again manually to generate a fresh signing secret.

When running in Jenkins, each of the three values generated by that script need to be created as "Secret text" credentials in Jenkins. Use the environment variable name as the credential ID and the long encoded string as the secret.

Quick-start for testing the build pipeline and the promote pipeline in development mode

Assuming that QUAY_IO_CREDS_USR, QUAY_IO_CREDS_PSW and MY_QUAY_USER environment variables are set and exported already.

export REKOR_HOST='' TUF_MIRROR=''

# if you have an RHTAP instance deployed including TPA
eval "$(hack/get-trustification-env.sh)"
# otherwise
# export FAIL_IF_TRUSTIFICATION_NOT_CONFIGURED=false

(bash build-pipeline.sh && export MY_IMAGES_TO_VERIFY=$(hack/show-image-ref) && bash promote-pipeline.sh) | tee log.out

Note that the MY_IMAGES_TO_VERIFY is a workaround so that the image being verified is the image that was just built. This is only for development purposes. If MY_IMAGES_TO_VERIFY is not set then the image list will be produced by the gather-deploy-images script as per usual.

Configuring Jenkins

Binaries The agent machines running jenkins (or if on master, that machine will need to have binaries configured for the jenkins user running the pipelines)

These will be checked prior to allowing execution to proceed. If any binaries are missing, there will be an error message printed. Install the required binary and re-run the shell mode or the jenkins agent.

ENV vars:
OK: IMAGE_URL
OK: IMAGE
OK: QUAY_IO_CREDS_USR
OK: QUAY_IO_CREDS_PSW
OK: DISABLE_ACS
OK: GITOPS_AUTH_PASSWORD
OK: POLICY_CONFIGURATION
OK: REKOR_HOST
OK: IGNORE_REKOR
OK: INFO
OK: STRICT
OK: EFFECTIVE_TIME
OK: HOMEDIR
Binaries:
OK: git in /usr/bin/git
OK: curl in /usr/bin/curl
OK: jq in /usr/bin/jq
OK: yq in /usr/local/bin/yq
OK: buildah in /usr/bin/buildah
OK: syft in /mnt/g/wslbin/syft
OK: cosign in /usr/local/bin/cosign
OK: python3 in /usr/bin/python3
Env vars and binaries ok

The library requires some secrets to be defined in your Jenkins cluster

There are utility scripts in the hack directory which allow automated setting of the Jenkins Secrets. You will need local env vars, and your Jenkins access token configured. Run the script and it will tell you what you are missing. It will set the credentials required for the Jenkins pipeline to work.

If you have an RHTAP instance deployed including TPA, run this before running the jenkins-set-secrets script: eval "$(hack/get-trustification-env.sh)". This will set TRUSTIFICATION_* environment variables for you, which jenkins-set-secrets will then deploy to your Jenkins instance.

To set the secrets, run hack/jenkins-set-secrets To validate secrets are set run hack/jenkins-get-secrets

You can also check the credentials view in your Jenkins instance.

Setting up Jenkins jobs

When you create a Jenkins-based component through Developer Hub (see RHTAP-integrated mode), you get a repository with a Jenkinsfile. Import the repository into your Jenkins instance as follows:

1. In the Jenkins homepage, click + New Item
2. Create a project of type Pipeline. Give it the same name as the Developer Hub component name.
3. In the next view, in the Pipeline section, select Definition: Pipeline script from SCM, then select SCM: Git
4. Fill in your Repository URL, fix the Branch Specifier, save the pipeline
5. You can now run the pipeline manually

CI Testing

For ease if testing Jenkins, Gitlab CI and Github Actions, there is a script in the root called ci-test.sh. This script will setup and run the build and promotion pipelines in an automated way. The script will perform the following

1. Reset the build and gitops repositories for each of the CI systems to a base source and application sample, these repos are typically called tssc-dev-source-{ci-name} and tssc-dev-gitops-{ci-name} . Note github Actions may have a different name due to history. The ci-test.sh script will print the names of the repos used as well as any manual steps on creating repos (gitlab)
2. Copy the CI pipelines and env setup for the ci tests into the build and gitops repos. The script can be run with RHTAP connected or not installed. If running without an RHTAP (usefull for testing build, sbom, ec) the scripts will detect that there is no cluster and will disable ACS, REKOR and TRUSTIFICATION. If you have a running cluster, this can also be accomplished with oc logout for simpler testing.'
3. Kick off a build on each CI if manually required.
4. To get automated pull requests, run bash hack/wait-for-gitops-update.sh in a separate window before running the ci-test.sh. It will watch the three gitops repos for updates from build and automatically send a PR to update stage from dev. This can be used to validate a full build/promotion cycle before releasing into the tssc-sample-templates.
5. You can also run hack/rhtap-promote --repo repo-url to trigger a PR for an individual repo.

Gitlab CI, Github Action, Jenkins Configuration

ci-test will configure all three CI systems, specifically secrets for your configuration using the following scripts. When RHTAP auto-configures the CI systems secrets this will no longer be needed. These scripts could also be useful for customers. See the following scripts

hack/ghub-set-vars 
hack/glab-set-vars  
hack/jenkins-set-secrets