Pipeline Event Plugin Design

[JohnNiang]

Motivation

At present, the Devops controller realizes the data synchronization of PipelineRun by polling, which is very inefficient and wastes computing resources. If the PipelineRun status events can be sent on the Jenkins side, and DevOps apiserver can provide webhook to receive the corresponding events and trigger the update of PipelineRun status in time. This makes the status more real-time.

We have found a Jenkins plugin, Cloudevents Plugin for Jenkins. However, there are too many problems and they do not meet the needs of Pipeline plugins. Therefore, it is necessary to manually implement a Jenkins plugin.

Goals

• Support CasC configuration.
• Allows users to configure webhook address and corresponding secret.
• Allows the receiver to receive event content in a fixed format.
• Allows the receiver to receive the status change of Workflow Run.
• Allows the receiver to receive the execution status change of Workflow Step.

Non-Goals

• Provide events of cloudevent specification.
• Allows the receiver to receive the status change of Workflow Job and Freestyle project.

Use Cases

• When the Pipeline on Jenkins side is triggered passively, such as the Pipeline triggered after the first operation of multi-branch Pipeline or receiving webhook from GitLab or GitHub, send events to the third-party platform (like ks-devops), and it will synchronize the Pipeline status in time.
• When the build on Jenkins side is deleted, send an event to the third-party platform (like ks-devops), and it will delete the corresponding Pipeline in time.

Requirements

• Users need to install Workflow and Pipeline plugins. The Pipeline Event Plugin relies on them and mainly captures the events triggered by Workflow.
• Users need to create Pipeline and multi-branch Pipeline. As mentioned above, the Pipeline Event Plugin is mainly for Workflow events. If a user creates an Freestyle Job, he will not receive any events.

Design Draft

This document has been opened to begin a discussion around the feature. There is a draft implementation in Pipeline Event Plugin.

Pipeline Event Configuration

• Receiver URL(required): Target URL where event sent to
• Secret (optional): Uses to encrypt event data

Analysis of Pipeline

// Fire an event for Workflow started
pipeline {
    agent any

    stages {
        stage('Hello') {
            steps {
                echo 'Hello World' // Fire an event for atom step
                echo 'Hello ks-devops' // Fire an event for another atom step
            }
        }
    }
}
// Fire an event for Workflow completed

Examplary event body structure

The structure of the event body remains unchanged. The only change is the data field. Its structure will depend on the event type and data type.

{
  "type": "run.intialize",
  "source": "",
  "id": "A234-1234-1234",
  "time": "2018-04-05T17:31:00Z",
  "datatype": "WorkflowRun",
  "data": {
    "fullName": "my-devops-project/my-pipeline",
    "name": "my-pipeline",
    "number": 1,
    "queueId": 11,
    "isMultiBranch": false
  }
}

{
  "type": "run.running",
  "source": "",
  "id": "A234-1234-1234",
  "time": "2018-04-05T17:31:00Z",
  "datatype": "WorkflowRun",
  "data": {
    "fullName": "my-devops-project/my-pipeline",
    "name": "my-pipeline",
    "number": 1,
    "queueId": 11,
    "isMultiBranch": false,
    "state": "Running"
  }
}

{
  "type": "job.created",
  "source": "",
  "id": "A234-1234-1234",
  "time": "2018-04-05T17:31:00Z",
  "datatype": "FreestyleProject",
  "data": {
    "fullName": "my-devops-project/my-pipeline",
    "name": "my-pipeline"
  }
}

Examplary event data structure

• Workflow Run: onInitialize

  type: workflow.run.initialize

  {
    "fullName": "my-devops-project/my-pipeline",
    "name": "my-pipeline",
    "number": 1,
    "queueId": 11,
    "isMultiBranch": false
  }

  {
    "fullName": "my-devops-project/my-pipeline",
    "name": "my-pipeline",
    "number": 1,
    "queueId": 11,
    "isMultiBranch": true,
    "scmRefName": "main"
  }

• Workflow Run: onStarted

  type: workflow.run.started

  {
    "fullName": "my-devops-project/my-pipeline",
    "name": "my-pipeline",
    "number": 1,
    "queueId": 11,
    "isMultiBranch": false
  }

• Workflow Run: onCompleted

  type: workflow.run.completed

  {
    "fullName": "my-devops-project/my-pipeline",
    "name": "my-pipeline",
    "number": 1,
    "queueId": 11,
    "isMultiBranch": false
  }

• Workflow Run: onFinished

  type: workflow.run.finished

  {
    "fullName": "my-devops-project/my-pipeline",
    "name": "my-pipeline",
    "number": 1,
    "queueId": 11,
    "isMultiBranch": false
  }

• Flow Step: onStarted

  type: workflow.run.step.started

  {
    "fullName": "my-devops-project/my-pipeline",
    "name": "my-pipeline",
    "number": 1,
    "queueId": 11,
    "isMultiBranch": false,
  
    "stepId": 5
  }

Extensiblity

In the future, we are allowed to support more events, which requires our extensiblity. At present, we mainly design and develop Pipeline related events. If there are other event requrements in the future, we need to enable developers to better develop new events and enable users to easily receive new events.

Examplary Freestyle Job Event Development

• Freestyle Job: onCreated

  tpye: job.created

  {
    "fullName": "my-devops-project/my-pipeline",
    "parent": "my-devops-project",
    "name": "my-pipeline"
  }

TBDs

1. Minimum supported Jenkins version?

   Jenkins 2.277.4.

2. Minimum supported JDK version?

   JDK 8.

3. Do we need to support cloudevent specification now?

   Not yet now.

4. If the step is executed too fast and multiple events are sent in a short time, how do we deal it?

   Not yet now.

5. Do we need to ensure the sequence of events?

   Not yet now.

6. Do we need to support retry mechanism?

   Not yet now.