Snomed Jenkins - A fully automated CI/CD pipeline implementation

• Snomed Jenkins - A fully automated CI/CD pipeline implementation
• Summary
  • What each step does
• Useful URL's
• Configuration
  • Master Code Estate Spreadsheet
    • Spreadsheet column and in column delimiters
  • Environment variables
• Docker
• Architecture
  • Jenkins Pipelines
    • Build step:
    • Security step:
  • Shell Scripts
    • 000_Config.sh
• Jenkins jobs starting with underscore
  • _CleanSpreadsheetCache_
  • _DailyAnalysis_
  • _PipelineCreationJob_
    • jobMake.groovy
• How to test and update the scripts
• Jenkins Plugins
• Credentials
• Linux box libraries installed
• GITHUB authentication
• Useful/Common tasks
  • Allow plugins to provide images/fonts etc
  • How to update Jenkins version
  • How to update plugins

This project contains all code for the Jenkins build pipelines for Snomed CT, except for the pipelines and configuration, which are controlled by Jenkins itself for security.

This CI/CD process is designed from the ground up to by fully automated. The only action that needs to occur is maintenance of the Master Code Estate spreadsheet. This spreadsheet is all powerful and controls what is built and how.

This project is designed to be:

• Low cost (all open source).
• Easy to maintain.
• Fully automated.
• Fully self-managing.
• Consistency, every job type, even if there are thousands will run the same. All you need is a row in the master spreadsheet.
• Powerful.
• Efficient.
• Configurable.
• Feature rich.
• Modern.

The builds are controlled by several mechanisms working together (detailed below):

• Pipelines—located in Jenkins
• Groovy scripts
• Shell scripts

This results in a system, which:

• Creates and deletes the build projects in jenkins.
• Control what steps are run and when.
• Automatically create tickets in Jira for new CVE's that occur overnight.
• Automatically create and update GitHub build hooks, so that a git push will cause a build to run.
• Build maven, Gradle and UI projects.
• Push to DockerHub
• Push to Nexus
• Run Cypress tests and publish reports, images and video results of the tests.
• Collate various reports
• Manage branches, their creation and deletion.
• and so on.

Summary

In summary:

• Jenkins runs the jobs and contains configuration, secrets, and the pipeline definitions.
• This project is checkedout by the PipelineCreationJob project and, which runs various scripts, but mainly the scripts/jobMake.groovy.
• The scripts/jobMake.groovy script uses groovy to create and configure projects according to the Master Code Estate Spreadsheet.
• These jobs look after themselves.
• Each job runs within a pipeline, the pipeline contains a series of steps, the actions performed by a step and dictated by the shell scripts located in the scripts folder.

What each step does

Depending on the pipeline the shell scripts for each step do the following tasks.

• 000_Config.sh - Used by steps below to get config of project from spreadsheet.
• 010_Initialize.sh - Check gradle/maven version and output set.
• 020_SanityCheck.sh - Check for license file, if a maven project there is a pom, run gitleaks and so on.
• 500_Build.sh - Use the build tool maven/gradle to build the project.
• 510_Test.sh - Run unit tests.
• 600_Security.sh - Run OWASP dependency check.
• 610_Documentation.sh - Use Doxygen to generate code documentation.
• 620_Performance.sh - Run performance tests.
• 630_Quality.sh - Generate code coverage report with Jacoco.
• 640_EndToEndTest.sh - Run Cypress end to end tests.
• 650_Audit.sh - Run code quality with SonarQube.
• 660_Deploy.sh - Deploy to nexus and if the project is configured with maven/jib plugin to dockerhub.
  • Note that the deployment location depends on the branch name:
    • If branch name is master/main/release-candidate artifact is deployed to releases
    • If branch name is develop or ends in nexus the artifact is deployed to snapshots
  • Branches with any other name are NOT deployed to nexus.
• 800_TriggerDownstream.sh - Start any downstream projects.
• 900_Cleanup.sh - Save disc space on jenkins if needed.
• 950_SelectCommsChannel.sh - Library used by pipelines to send messages to the correct destination.

Useful URL's

Common/useful URL's these will work for local docker installations.

• Replace http://localhost:9000 with the URL of your sonar server.
• Replace http://localhost:8083 with the URL of your jenkins server.

Description	URL
http://localhost:9000/	Sonar Qube
http://localhost:8083/	Jenkins
http://localhost:8083/manage/	Jenkins management screens.
http://localhost:8083/manage/configureTools/	Tools configuration, git, java etc.
http://localhost:8083/configure	This URL is used for configuring the Jenkins environment.
http://localhost:8083/computer	This URL shows the agents that are connected to the Jenkins system.
http://localhost:8083/restart	This URL is used to restart Jenkins when it is safe to do so. Jenkins will not restart until running jobs are complete. This requires the "Safe Restart" plugin.
http://localhost:8083/safeRestart	This URL is used to safely restart Jenkins in a manner similar to /restart. It also requires the "Safe Restart" plugin.
http://localhost:8083/quietDown	This URL is used to put Jenkins in a "Quiet Down" mode. In this mode, no new build jobs are started.
http://localhost:8083/cancelQuietDown	Stops quiet down mode
http://localhost:8083/exit	Forces Jenkins to quit. Use with caution as it does not wait for build jobs to complete and may result in data loss.
http://localhost:8083/script	This URL is for running Groovy scripts inside Jenkins, often used for administrative purposes.
http://localhost:8083/pipeline-syntax	Jenkins Pipeline Syntax
http://localhost:8083/plugin/job-dsl/api-viewer/index.html	Jenkins Documentation
http://localhost:8083/safeExit	Jenkins safe exit

Configuration

Configuration for the build process is stored in two locations:

Master Code Estate Spreadsheet

The primary spreadsheet controls, which jobs are built and how. This spreadsheet contains a list of projects, 1 per line. The columns define the information for the project, which inturn dictatates how the project is build. The column headings are (note these are EXACT NAMES AND ORDER MUST NOT BE CHANGED as they are read by 000_Config and jobMake.groovy):

• Jenkins Build Enabled
• Project Name
• GroupId:ArtifactID
• Build Tool
• Language
• Project Type
• Deploy Enabled
• Slack Channel
• Notified Users
• Uses BOM?
• Snomed Dependencies
• Owner
• Notes

The master code estate spreadsheet is stored in the Jenkins environment variable SNOMED_SPREADSHEET_URL i.e. something like https://docs.google.com/spreadsheets/SOME_GOOGLE_DOC_ID.

BE CAREFUL EDITING THIS SPREADSHEET

For example, DO NOT edit the column titles/order etc., it will break the build process. Google does keep an edit history, so you can always rewind any changes (and find who broke the spreadsheet).

Spreadsheet column and in column delimiters

The master spreadsheet is used by groovy and bash files. This limits the complexity of parsing the spreadsheet. So simple string splitting is used. To simplify this tabs are used in dividing the columns. Note the "Notified Users" and "Snomed Dependencies" columns can be further divided, these are split internally with the pipe symbol.

Environment variables

The environment variables are located here: https://YOUR_JENKINS_HOST/manage/configure - access is restricted.

Name	Value	Description
DOWNSTREAM_ENABLED	true/false	Downstream spawning of tasks
LICENSE_EXPECTED_CHECK_SUM	md5 checksum of your LICENSE.md files.	Checksum of generic LICENSE file, without year line.
SCRIPTS_PATH	/var/lib/jenkins/workspace/_PipelineCreationJob_/scripts	Where the scripts can be found.
SNOMED_SPREADSHEET_URL	https://docs.google.com/spreadsheets/SOME_GOOGLE_DOC_ID	Where the master code spreadsheet is located.
SNOMED_TOOLS_URL	SOME_URL	Used to link systems together.
SONAR_URL	https://YOUR_SONAR_HOST/	SonarQube host URL.
VERBOSE	true/false	Slightly more output from jobs, for example see 010_Initialize.sh

Docker

This project contains a subfolder for a local installation of Jenkins and SonarQube. This enables you to run Jenkins locally to perform builds and test any changes you may wish to try. Find the documentation in this README.md.

Architecture

The system consists of a standard Jenkins installation, with some extra plugins see the plugins section

We have tried to keep the use of groovy to a minimum, all build steps are simple, short bash files, which do one step within a pipeline; these are located in the scripts folder.

Jenkins Pipelines

The builds are controlled by pipelines:

There are several pipelines, the main one being SnomedPipeline_Maven_Jdk17 which is the pipeline executed for java 17 maven project (as specified in the Master Spreadsheet)

Each pipeline step decides what to run and what the steps are. These can be seen in any build and look like this.

The pipelines themselves have as little code as possible (they are a Groovy DSL). The code for each step is located in the Shell Scripts, see section.

Build step:

• Shell scripts starting with a number are called from the pipelines roughly in numerical order.
  • 000_Config.sh is an exception see this section
• The remaining shell scripts are part of the normal build process.
• The groovy scripts are all part of the control of Jenkins using groovy and the DSL: https://plugins.jenkins.io/job-dsl/

Within the pipelines there are a series of steps. For example the pipelines build step looks like this.

So this step simply runs the shell script 500_Build.sh always.

stage('Build') { steps { sh "../_PipelineCreationJob_/scripts/500_Build.sh" } }

Security step:

Here is a more complex step. There are 3 pieces of logic:

1. when/anyOf: It only runs this for the branches listed, i.e. main/master/develop/release-candidate.
2. steps: What to run, in this case the shell script 600_Security.sh.
3. post: On success publish a report.

Also since for this particular scenario, we do not want the build to stop on failure there are two extra arguments/code:

1. For the steps we have 600_Security.sh || true. This is shell magic for always return true even if the shell script fails.
2. For the post/success for extra protection with have stopBuild: false.

stage('Security') {
    when {
        anyOf {
            branch 'main'
            branch 'master'
            branch 'develop'
            branch 'release-candidate'
        }
    }
    steps {
        sh "../_PipelineCreationJob_/scripts/600_Security.sh || true"
    }
    post {
        success {
            dependencyCheckPublisher(stopBuild: false)
        }
    }
}

Shell Scripts

Within the pipelines there are a series of steps, most of the run a single shell script. These are in the scripts folder. Most of these scripts are of the form, decide what the projects and steps characteristics are and then run the appropriate commands. For example for 500_Build.sh looks like this (at the time of writing).

#!/usr/bin/env bash                                       # Its a bog standard bash shell script.
source "$SCRIPTS_PATH/000_Config.sh"                      # Loads the configuration from the master code spreadsheet.
figlet -w 500 "Build"                                     # Print big obvious banner in the log.
 
case $SNOMED_PROJECT_LANGUAGE in                          # Decide which language we are building.
    Cypress)                                              # Language is Cypress so use npm and ng to build.
        npm install
        ng build
        ;;
    *)                                                    # Here onwards, if the build tool is Maven use that, if Gradle use that, if none then echo a message, anything else fail the build.
        case $SNOMED_PROJECT_BUILD_TOOL in
            maven)
                mvn -U clean package -DskipTests -Ddependency-check.skip=true
                ;;
            gradle)
                ./gradlew clean build buildDeb -x test -x spotbugsMain -x spotbugsTest -x checkstyleTest -x checkstyleMain
                ;;
            none)
                echo "No build tool required."
                ;;
            *)
                echo "Unknown build tool: ${SNOMED_PROJECT_BUILD_TOOL}"
                exit 1
                ;;
        esac
    ;;
esac

The shell scripts are numbered from 010 onwards, representing the rough order they are run, and the first number representing the section the script belongs in, broadly speaking.

000_Config.sh

The 000_Config.sh script provides a mechanism for all of the shell scripts to have access to the configuration spreadsheet. To do this it:

• Downloads the spreadsheet (there is a simple cache mechanism in place)
• Parses the spreadsheet and assigns a series shell variables which other scripts can use.
  • So from the spreadsheet the name column will be assigned to the variable $SNOMED_PROJECT_NAME

Jenkins jobs starting with underscore

All jobs starting with an underscore are special. The underscore protects them from being deleted for easy identification and auto-management. These jobs have to be manually created and configured, all other jobs are created/delete automatically.

ALL OTHER JOBS ON THE SYSTEM ARE AUTOMATICALLY MANAGED/DELETED - SO IF YOU EDIT THEM YOU WILL LOSE YOUR CHANGES THE NEXT TIME THE MANAGEMENT JOB RUNS.

_CleanSpreadsheetCache_

This a convenience job which does the following:

• Removes the cached/downloaded spreadsheet file, this is useful to makesure latest spreadsheet changes are loaded:
• Uses the linux set command to output all environment variables, this can be useful for debugging.

This job is run manually as and when required.

_DailyAnalysis_

This job runs on a schedule. It does the following:

# Search for CVE's in project reports and generate tsv file.
../_PipelineCreationJob_/jobMakeCveTable.sh tsv

# Analyse project pom.xml files and generate PNG and SVG images.
../_PipelineCreationJob_/jobMakeDependencyGraph.sh

# Create new Jira tickets if required.
../_PipelineCreationJob_/createCveJiraTickets.sh

# Use TSV file and existing Jira tickets to create HTML report.
../_PipelineCreationJob_/jobMakeCveTable.sh html

_PipelineCreationJob_

This job runs on a schedule (or if you update the spreadsheet you can run it manually). It runs 3 scripts:

• jobMake.groovy - See below.
• approveAllScripts.groovy - There is a security mechanism in Jenkins, this is needed to approve scripts automatically.
• jobMakeCreateGithubWebHook.sh - This uses the github API to make build webhooks for the projects if needed in github.

jobMake.groovy

The creation of the pipelines is automated for each project. This is performed by the largest/most complex part of this snomed-jenkins project.

https://github.com/IHTSDO/snomed-jenkins/blob/main/scripts/jobMake.groovy

This script is written in Groovy, again with some DSL extensions provided by the Jenkins Job DSL plugin. The pseudo code for the script is:

Download the configuration spreadsheet

for each row of the spreadsheet
  Call makeJobs, if the project is enabled

function makeJob:
  Add information to a hook file (used to make the build hooks in github)
  Create a pipeline for the project
  Create a nightly security check job for appropriate projects
  Create a nightly e2e job for appropriate projects

How to test and update the scripts

If you update a script you can test it by updating the corresponding script on the jenkins box, bear-in-mind that any updates will be removed when this task next updates.

The easiest was is to upate within this project, commit and then run the _PipelineCreationJob_, this will download the latest scripts and when a pipeline is run your new code will be executed.

Jenkins Plugins

The following is a list of the plugins we use in our Jenkins instance.

• AnsiColor
• Ant
• Config file provider
• Doxygen
• Gradle
• Groovy
• HTML Publisher
• JaCoCo
• Job DSL
• Maven Integration
• OWASP Dependency-Check
• Pipeline
• Dashboard ViewVersion
• Multibranch Scan Webhook Trigger
• Theme
  • http://afonsof.com/jenkins-material-theme/
  • https://devopscube.com/setup-custom-materialized-ui-theme-jenkins/#:~:text=Uploading%20Custom%20CSS%20TO%20Jenkins%20Server&text=Step%201%3A%20Login%20to%20your,layout%20inside%20the%20userContent%20directory.&text=Step%203%3A%20cd%20into%20the,css%20file.
  • Jenkins simple theme plugin
  • Dark Theme
  • Material Theme
  • Set CSS TO this:

.logo img {
    content: url(/userContent/layout/logo.png);
}

#jenkins-name-icon {
    display: none;
}

.logo:after {
    content: 'Snomed Jenkins Dev Server';
    font-size: 35px;
    font-family: Arial, Helvetica, sans-serif;
    margin-left: 20px;
    margin-right: 12px;
    line-height: 40px;
}

Credentials

• These are all setup here: https://YOUR_JENKINS_HOST/manage/credentials/ - access is restricted.
• You can see in the pipelines how these are passed
• The shell scripts and pipelines use these secrets to perform their tasks.

Linux box libraries installed

I recommend installing cht and tldr. These are useful to get simple pages and examples of the following commands.

• JDK* - 11 and 17 for example, depending on your source.
• Also, the usual tools, git, maven, gradle, groovy, curl, wget, jq and imagemagick.
• doxygen - javadoc++
• figlet - Print a nicer banner.
• graphviz/dot - Used to convert dot files to images.
• bc - needed in bash to compare floating point numbers.
• jq - query json from the command line.
• xmlstarlet - query xml from command line.
• xmllint (installed in libxml2-utils) - query xml from command line.
• nodejs, npm and npx with:
  • sudo tar -C /usr/local --strip-components 1 -xvf node-v20.9.0-linux-x64.tar.xz
• xfvb Virtual graphics buffer.
• libgbm-dev generic buffer management API

For more information see the docker install/setup script here.

GITHUB authentication

Github requires a key for access, this can be created as follows:

Note this is performed on the machine you are connecting from, i.e. docker container or your machine.

• Create the key pair using:

`ssh-keygen -t ed25519`

• This will create some files in your ~/.ssh folder: id_ed25519 and id_ed25519.pub.
• Add the public key to github.
• Copy both files to the .ssh folder on jenkins.
• To test access do the following:

mkdir tmpfolder
cd tmpfolder
git clone [email protected]:IHTSDO/snomed-jenkins.git
cd ..
rm -ef tmpfolder

• This will test repo download but also add/create a ~/.ssh/known_hosts file.

Jenkins should now have access to github.

Useful/Common tasks

Allow plugins to provide images/fonts etc

Jenkins will block some content from being shared, you can tell this as images, icons pages may be missing. If this happens you may need to allow this content to be served from Jenkins. To allow this go do the script console: http://localhost:8083/manage/script Paste the following snippet and run it.

System.setProperty("hudson.model.DirectoryBrowserSupport.CSP",
"""
    default-src * 'unsafe-inline';
    script-src  * 'unsafe-inline' 'unsafe-eval';
    connect-src * 'unsafe-inline';
    font-src    * data: blob: 'unsafe-inline';
    img-src     * data: blob: 'unsafe-inline';
    child-src   * data: blob: 'unsafe-inline';
    style-src   * 'unsafe-inline';
    object-src  * 'unsafe-inline';
""")

For more information see: https://www.jenkins.io/doc/book/security/configuring-content-security-policy/

How to update Jenkins version

Ask DevOps to do this the download and instructions are here: https://www.jenkins.io/download/

How to update plugins

To update the plugins on jenkins:

1. Visit: http://localhost:8083/manage/pluginManager/
2. Select all the plugins you want to update.
3. Click update and the restart link.
4. You may need to manually restart jenkins using this link http://localhost:8083/restart, or on rare occasions ask DevOps to restart the machine.