Skip to content

A Proof of Concept of codecentric/jenkins helm chart with helm 3, Jenkins Configuration as Code and Digital Ocean Kubernetes.

License

Notifications You must be signed in to change notification settings

nubenetes/jenkins-CasC-kubernetes-demo

Repository files navigation

Proof of Concept: Jenkins Configuration as Code on Kubernetes. A Codecentric/Jenkins Helm 3 Sample Chart on Digital Ocean Kubernetes with Spring Petclinic Demo Pipeline

Introduction

Disclaimer

  • These sample configuration files may not be optimized for your environment and can be significantly improved.

Requirements

  • A registered domain name publicly reachable from Internet. I have a registered domain for testing purposes.
  • A Digital Ocean Kubernetes Cluster (kubernetes 1.16.8+) with 1 worker node with 2 VCPU and 4GB RAM.
  • A GitHub private repository with the content of this repo, containing sensitive data like usernames and passwords.
  • A DockerHub private repository.
  • Helm 3 instead of helm 2. Simpler and with improved security (tiller is not required).
  • Apply the following Helm 3 Charts on DO Kubernetes (more details below):

Tips for validating JCasC YAML

  • Set up jenkins via User Interface and export the running configuration to a YAML file. This feature is provided by Jenkins Configuration as Code Plugin. This YAML file cannot be imported afterwards, but it will help you to figure out what configuration lines should be added to JCasC YAML.
  • Use Visual Studio Jenkins JCasC-Plugin This extension is used to integrate a live jenkins instance configuration with your editor. It can be used to edit and validate YAML files.

Tips for adding jenkins jobs to JCasC. Use a Seed Job

  • Use 1 file per jenkins job that needs to be configured as source code. Change configuration by adding 1 file per pipeline. Management of pipelines by adding files instead of having 1 single file with all the configurations. Otherwise a single error would break all the pipelines and troubleshooting would be harder.
  • Use a Jenkins Seed Job that loads all the Jenkins Jobs (1 file per jenkins job):
    • This seed job is usually written with JobDSL and can be set up in JCasC YAML or be imported manually as jenkins freestyle job.
    • Use regular expressions in the seed job to select and filter out all the Jenkinsfiles and/or JobDSL files that describe your jenkins jobs (pipelines).

Screenshots

Environment

This proof of concept was run with with the following releases:

YAML Settings

  1. Please do a quick search of <my_ string to identify the settings that need to be updated accordingly in your specific environment:
$ grep -ri \<my_ ./*.yaml
./deployment.yaml:        image: <my_dockerhub_username>/spring-petclinic:latest
./externaldns-values.yaml:  apiToken: <my_digitalocean_api_token>
./externaldns-values.yaml:domainFilters: [ '<my_public_dns_domain.com>' ]
./ingress.yaml:  - host: petclinic.<my_public_dns_domain.com>
./jenkins-values.yaml:    - jenkins.<my_public_dns_domain.com>
./jenkins-values.yaml:    ADMIN_USER: <my_jenkins_admin_username>
./jenkins-values.yaml:    ADMIN_PASSWORD: <my_jenkins_admin_password>
./jenkins-values.yaml:      username: <my_dockerhub_username>
./jenkins-values.yaml:      password: <my_dockerhub_password>
./jenkins-values.yaml:                    id: "github_<my_github_username>"
./jenkins-values.yaml:                    passphrase: "<my_github_password>"
./jenkins-values.yaml:                          <my_private_key> 
./jenkins-values.yaml:                    username: "<my_github_username>"
./jenkins-values.yaml:                              certificate-authority-data: <my_k8s_certificate-authority-data>
./jenkins-values.yaml:                              server: https://<my_k8s_server_id_in_kube_config>.k8s.ondigitalocean.com
./jenkins-values.yaml:                            name: <my_k8s_name_in_kube_config>    
./jenkins-values.yaml:                              cluster: <my_k8s_name_in_kube_config>
./jenkins-values.yaml:                              user: <my_k8s_name_in_kube_config>-admin
./jenkins-values.yaml:                            name: <my_k8s_name_in_kube_config>
./jenkins-values.yaml:                          current-context: <my_k8s_name_in_kube_config>
./jenkins-values.yaml:                          - name: <my_k8s_user_name_in_kube_config>-admin
./jenkins-values.yaml:                              token: <my_k8s_user_token_in_kube_config>
./jenkins-values.yaml:              url: "https://jenkins.<my_public_dns_domain.com>"
./jenkins-values.yaml:                            url '[email protected]:<my_github_username>/helm-charts-do.git' 
./jenkins-values.yaml:                            credentials 'github_<my_github_username>'
./jenkins-values.yaml:                            url '[email protected]:<my_github_username>/helm-charts-do.git' 
./jenkins-values.yaml:                            credentials 'github_<my_github_username>'
  1. Replace each setting with your own parameter. For example:
  • Replace <my_public_dns_domain.com> with domain-example.com . I set up a registered domain name that I have for testing purposes.
  • Replace <my_jenkins_admin_password> with your_own_jenkins_admin_password
  • <my_digitalocean_api_token> : Digital Ocean API Token
  • Fill in some of the required parameters from your DO kubernetes settings available in $HOME/.kube/config.
  • etc

Settings to be updated on each DO Kubernetes Cluster

  • Each new DO Kubernetes cluster (and recreation) requires to update jenkins-values.yaml in order to authenticate against each specific cluster.
  • These are the three parameters to update in "kubeconfig: id: digitalocean" section within jenkins/values.yaml:
    1. certificate-authority-data: <my_k8s_certificate-authority-data>
    2. server: https://<my_k8s_server_id_in_kube_config>.k8s.ondigitalocean.com
    3. token: <my_k8s_user_token_in_kube_config>

Settings to be updated on Petclinic Software Delivery Demo Pipeline

  • Remember to check and maintain the following environment var in petclinic.Jenkinsfile to match the SpringBoot release specified in petclinic's parent POM file (Upgrade to Spring Boot 2.3.0.RC1).
  • For example:
    • spring_boot_release = '2.3.0.BUILD-SNAPSHOT'

Setting up Nginx Ingress Controller

  • Several options: with helm and without helm (see below refs).
  • With Helm 3:
$ helm install nginx-ingress stable/nginx-ingress --set controller.publishService.enabled=true
NAME: nginx-ingress
LAST DEPLOYED: Wed Jan  1 18:34:02 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w nginx-ingress-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

Setting up External Dns (included in deploy.sh)

  • Kubernetes' external-dns plugin is required.
  • This step is already included in deploy.sh script.
  • Updating your hosts file (windows, WSL or similar) should not be required.
  • Be aware the exposed IP address changes each time DO Kubernetes is recreated.
~/helm$ helm search repo external-dns
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                       
stable/external-dns     2.13.0          0.5.17          ExternalDNS is a Kubernetes addon that configur...
~/helm$ helm install external-dns stable/external-dns -f externaldns-values.yaml
NAME: external-dns
LAST DEPLOYED: Fri Dec 14 10:28:35 2019
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

To verify that external-dns has started, run:

kubectl --namespace=default get pods -l "app.kubernetes.io/name=external-dns,app.kubernetes.io/instance=external-dns"

Deployment with deploy.sh

Run ./deploy.sh script.

Uninstall with uninstall.sh

Run ./uninstall.sh script.

Helm Commands of Interest

helm pull codecentric/jenkins --version 1.6.0
helm pull codecentric/jenkins --verify --version 1.6.0

Troubleshooting

kubectl logs -f jenkins-656b5fccc7-pv6f8 | egrep -i '(error|failure|exception|volume|claim|warning)' --color

References

Jenkins Configuration as Code

Visual Studio Extensions for Jenkins

Codecentric Jenkins Helm Chart

Digital Ocean Kubernetes

Declarative Pipelines DSL in Jenkins

Job DSL

Jenkins Seed Job

Jenkins Kubernetes Continuous Deploy Plugin

SpringBoot Docker

Maven

Petclinic

Petlinic kubernetes

Petclinic on GKE

About

A Proof of Concept of codecentric/jenkins helm chart with helm 3, Jenkins Configuration as Code and Digital Ocean Kubernetes.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published