Proof of Concept: Jenkins Configuration as Code on Kubernetes. A Codecentric/Jenkins Helm 3 Sample Chart on Digital Ocean Kubernetes with Spring Petclinic Demo Pipeline
- Introduction
- Disclaimer
- Requirements
- Tips for validating JCasC YAML
- Tips for adding jenkins jobs to JCasC. Use a Seed Job
- Screenshots
- Environment
- YAML Settings
- Setting up Nginx Ingress Controller
- Setting up External Dns (included in deploy.sh)
- Deployment with deploy.sh
- Uninstall with uninstall.sh
- Helm Commands of Interest
- Troubleshooting
- References
- This is a Proof of Concept of Codecentric/Jenkins Helm Chart with Helm 3, Jenkins Configuration as Code and Digital Ocean Kubernetes.
- This solution is not based on Kubernetes Operators.
- A sample of Software Delivery Pipeline is included with Spring Petclinic Demo and Jenkins Pipeline.
- This is 100% automated Jenkins on Kubernetes demo with PetClinic Build & Deploy (monolithic).
- These sample configuration files may not be optimized for your environment and can be significantly improved.
- A registered domain name publicly reachable from Internet. I have a registered domain for testing purposes.
- A Digital Ocean Kubernetes Cluster (kubernetes 1.16.8+) with 1 worker node with 2 VCPU and 4GB RAM.
- A GitHub private repository with the content of this repo, containing sensitive data like usernames and passwords.
- A DockerHub private repository.
- Helm 3 instead of helm 2. Simpler and with improved security (tiller is not required).
- Apply the following Helm 3 Charts on DO Kubernetes (more details below):
- Set up jenkins via User Interface and export the running configuration to a YAML file. This feature is provided by Jenkins Configuration as Code Plugin. This YAML file cannot be imported afterwards, but it will help you to figure out what configuration lines should be added to JCasC YAML.
- Use Visual Studio Jenkins JCasC-Plugin This extension is used to integrate a live jenkins instance configuration with your editor. It can be used to edit and validate YAML files.
- Use 1 file per jenkins job that needs to be configured as source code. Change configuration by adding 1 file per pipeline. Management of pipelines by adding files instead of having 1 single file with all the configurations. Otherwise a single error would break all the pipelines and troubleshooting would be harder.
- Use a Jenkins Seed Job that loads all the Jenkins Jobs (1 file per jenkins job):
- This seed job is usually written with JobDSL and can be set up in JCasC YAML or be imported manually as jenkins freestyle job.
- Use regular expressions in the seed job to select and filter out all the Jenkinsfiles and/or JobDSL files that describe your jenkins jobs (pipelines).
This proof of concept was run with with the following releases:
- codecentric/jenkins Helm Chart version 1.6.0
- helm 3.0.2
- jenkins/jenkins:2.222.3-alpine docker Image
- sprint-petclinic commit 6a18eec (May 2020)
- Digital Ocean Kubernetes 1.16.8
- stable/external-dns 2.13.0 helm chart (currently deprecated). Use bitnami/external-dns instead
- Please do a quick search of <my_ string to identify the settings that need to be updated accordingly in your specific environment:
$ grep -ri \<my_ ./*.yaml
./deployment.yaml: image: <my_dockerhub_username>/spring-petclinic:latest
./externaldns-values.yaml: apiToken: <my_digitalocean_api_token>
./externaldns-values.yaml:domainFilters: [ '<my_public_dns_domain.com>' ]
./ingress.yaml: - host: petclinic.<my_public_dns_domain.com>
./jenkins-values.yaml: - jenkins.<my_public_dns_domain.com>
./jenkins-values.yaml: ADMIN_USER: <my_jenkins_admin_username>
./jenkins-values.yaml: ADMIN_PASSWORD: <my_jenkins_admin_password>
./jenkins-values.yaml: username: <my_dockerhub_username>
./jenkins-values.yaml: password: <my_dockerhub_password>
./jenkins-values.yaml: id: "github_<my_github_username>"
./jenkins-values.yaml: passphrase: "<my_github_password>"
./jenkins-values.yaml: <my_private_key>
./jenkins-values.yaml: username: "<my_github_username>"
./jenkins-values.yaml: certificate-authority-data: <my_k8s_certificate-authority-data>
./jenkins-values.yaml: server: https://<my_k8s_server_id_in_kube_config>.k8s.ondigitalocean.com
./jenkins-values.yaml: name: <my_k8s_name_in_kube_config>
./jenkins-values.yaml: cluster: <my_k8s_name_in_kube_config>
./jenkins-values.yaml: user: <my_k8s_name_in_kube_config>-admin
./jenkins-values.yaml: name: <my_k8s_name_in_kube_config>
./jenkins-values.yaml: current-context: <my_k8s_name_in_kube_config>
./jenkins-values.yaml: - name: <my_k8s_user_name_in_kube_config>-admin
./jenkins-values.yaml: token: <my_k8s_user_token_in_kube_config>
./jenkins-values.yaml: url: "https://jenkins.<my_public_dns_domain.com>"
./jenkins-values.yaml: url '[email protected]:<my_github_username>/helm-charts-do.git'
./jenkins-values.yaml: credentials 'github_<my_github_username>'
./jenkins-values.yaml: url '[email protected]:<my_github_username>/helm-charts-do.git'
./jenkins-values.yaml: credentials 'github_<my_github_username>'
- Replace each setting with your own parameter. For example:
- Replace <my_public_dns_domain.com> with domain-example.com . I set up a registered domain name that I have for testing purposes.
- Replace <my_jenkins_admin_password> with your_own_jenkins_admin_password
- <my_digitalocean_api_token> : Digital Ocean API Token
- Fill in some of the required parameters from your DO kubernetes settings available in $HOME/.kube/config.
- etc
- Each new DO Kubernetes cluster (and recreation) requires to update jenkins-values.yaml in order to authenticate against each specific cluster.
- These are the three parameters to update in "kubeconfig: id: digitalocean" section within jenkins/values.yaml:
- certificate-authority-data: <my_k8s_certificate-authority-data>
- server: https://<my_k8s_server_id_in_kube_config>.k8s.ondigitalocean.com
- token: <my_k8s_user_token_in_kube_config>
- Remember to check and maintain the following environment var in petclinic.Jenkinsfile to match the SpringBoot release specified in petclinic's parent POM file (Upgrade to Spring Boot 2.3.0.RC1).
- For example:
- spring_boot_release = '2.3.0.BUILD-SNAPSHOT'
- Several options: with helm and without helm (see below refs).
- With Helm 3:
$ helm install nginx-ingress stable/nginx-ingress --set controller.publishService.enabled=true
NAME: nginx-ingress
LAST DEPLOYED: Wed Jan 1 18:34:02 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w nginx-ingress-controller'
An example Ingress that makes use of the controller:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: example
namespace: foo
spec:
rules:
- host: www.example.com
http:
paths:
- backend:
serviceName: exampleService
servicePort: 80
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.com
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls
- Kubernetes' external-dns plugin is required.
- This step is already included in deploy.sh script.
- Updating your hosts file (windows, WSL or similar) should not be required.
- Be aware the exposed IP address changes each time DO Kubernetes is recreated.
~/helm$ helm search repo external-dns
NAME CHART VERSION APP VERSION DESCRIPTION
stable/external-dns 2.13.0 0.5.17 ExternalDNS is a Kubernetes addon that configur...
- stable/external-dns 2.13.0 helm chart is currently deprecated. Use bitnami/external-dns instead
~/helm$ helm install external-dns stable/external-dns -f externaldns-values.yaml
NAME: external-dns
LAST DEPLOYED: Fri Dec 14 10:28:35 2019
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **
To verify that external-dns has started, run:
kubectl --namespace=default get pods -l "app.kubernetes.io/name=external-dns,app.kubernetes.io/instance=external-dns"
Run ./deploy.sh script.
Run ./uninstall.sh script.
helm pull codecentric/jenkins --version 1.6.0
helm pull codecentric/jenkins --verify --version 1.6.0
kubectl logs -f jenkins-656b5fccc7-pv6f8 | egrep -i '(error|failure|exception|volume|claim|warning)' --color
- Configure a CI/CD pipeline with Jenkins on Kubernetes
- Kubernetes plugin for Jenkins
- Kubernetes plugin Pipeline examples
- kubernetes-credentials-provider-plugin
- Jenkins JCasC-Plugin This extension is used to integrate a live jenkins instance configuration with your editor. It can be used to edit and validate YAML files.
- Jenkins Pipeline Linter Connector Validates Jenkinsfiles by sending them to the Pipeline Linter of a Jenkins server.
- secanis.ch: Jenkinsfile Support Adds syntax highlighting support for Jenkinsfile's. In this version, it's the same like Groovy is.
- ivory-lab: JenkinsFile Support Extension provides basic jenkinsfile support (highlighting, snippets and completion)
- JM Meessen: Declarative Jenkinsfile Support Adds syntax highlighting support for the declarative Jenkinsfile format flavour.
- Alessandro Fragnani: Jenkins Status
- Tips for declarative pipelines in Jenkins
- Where to put the wrapper for ansiColor Jenkins plugin in Jenkins Pipeline?
- Jenkins pipeline ansicolor console output
- Jenkins Pipeline Examples
- How to Seed Jenkins Build Jobs
- Tutorial Using the Jenkins Job DSL. Creating the Seed Job
- What are seed jobs in Jenkins and how does it work?
- Jenkins Tutorial: Implementing a Seed Job
- Setting up a shared library and seed job in Jenkins - Part 1
- Jenkins Jobs as Code with Groovy DSL
- Maven Wrapper The easiest way to integrate Maven into your project!
- Maven Plugins
- simple-java-maven-app For an introductory tutorial on how to use Jenkins to build a simple Java application with Maven.
- spring-petclinic.github.io
- A Dockerfile for Maven-based Github projects π
- Building Spring Docker Images π
- A Dockerfile for Maven-Based GitHub Projects ππ
- Jenkins CI reference pipeline for Java Spring Boot projects with Maven lifecycle and Docker packaging