Testing-events proposal

README.md

@@ -82,6 +82,10 @@ Handling Events associated with Continuous Deployment activities
 
 Handling Events associated with the health of the services deployed and running in a specific environment
 
+### [Testing Events](./testing-events.md)
+
+Handling Events associated with Test execution performed independently or as part of CI/CD pipelines.
+
 ### [CloudEvents Binding and Transport](./cloudevents-binding.md)
 
 Defining how CDEvents are mapped to CloudEvents for transportation and delivery

continuous-integration.md

@@ -19,10 +19,10 @@ This specification defines three subjects in this stage: `builds`, `artifacts` a
 | Subject | Description | Predicates |
 |---------|-------------|------------|
 | [`build`](#build) | A software build | [`queued`](#build-queued), [`started`](#build-started), [`finished`](#build-finished)|
-| [`testCase`](#testcase) | A software test case | [`queued`](#testcase-queued), [`started`](#testcase-started), [`finished`](#testcase-finished)|
-| [`testSuite`](#testsuite) | A collection of test cases | [`started`](#testsuite-started), [`finished`](#testsuite-finished)|
 | [`artifact`](#artifact) | An artifact produced by a build | [`packaged`](#artifact-packaged), [`published`](#artifact-published)|
 
+> `testCase`/`testSuite` events have moved to their own top-level bucket [Testing Events](testing-events.md)
+
 ### `build`
 
 A `build` is a process that uses a recipe to produce an artifact from source code.
@@ -31,35 +31,11 @@ __Note:__ The data model for `builds`, apart from `id` and `source`, only includ
 
 | Field | Type | Description | Examples |
 |-------|------|-------------|----------|
-| id    | `String` | See [id](spec.md#id-subject)| `1234`, `maven123`, `builds/taskrun123` |
-| source | `URI-Reference` | See [source](spec.md#source-subject) | `staging/tekton`, `tekton-dev-123`|
+| id    | `String` | Uniquely identifies the subject within the source. | `1234`, `maven123`, `builds/taskrun123` |
+| source | `URI-Reference` | [source](../spec.md#source) from the context | `staging/tekton`, `tekton-dev-123`|
 | type | `String` | See [type](spec.md#type-subject) | `build` |
 | artifactId | `String` | Identifier of the artifact produced by the build | `pkg:oci/myapp@sha256%3A0b31b1c02ff458ad9b7b81cbdf8f028bd54699fa151f221d1e8de6817db93427`, `pkg:golang/mygit.com/myorg/myapp@234fd47e07d1004f0aed9c` |
 
-### `testCase`
-
-A `testCase` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCase`  is the smallest unit of testing that the user wants to track. `testCases` are executed, and `testSuites` are for grouping purposes. For this reason, `testCases` can be queued.
-
-__Note:__ The data model for `testCase` only includes `id` and `source`, inputs and outputs of the process are not specified yet, as well as the relation to `testSuite`.
-
-| Field | Type | Description | Examples |
-|-------|------|-------------|----------|
-| id    | `String` | See [id](spec.md#id-subject)| `unitest-abc`, `e2e-test1`, `scan-image1` |
-| source | `URI-Reference` | See [source](spec.md#source-subject) | `staging/tekton`, `tekton-dev-123`|
-| type | `String` | See [type](spec.md#type-subject) | `testCase` |
-
-### `testSuite`
-
-A `testSuite` represents a set of one or more `testCases`.
-
-__Note:__ The data model for `testSuite` only includes `id` and `source`, inputs and outputs of the process are not specified yet, as well as the relation to `testCase`.
-
-| Field | Type | Description | Examples |
-|-------|------|-------------|----------|
-| id    | `String` | See [id](spec.md#id-subject)| `unit`, `e2e`, `security` |
-| source | `URI-Reference` | See [source](spec.md#source-subject) | `staging/tekton`, `tekton-dev-123`|
-| type | `String` | See [type](spec.md#type-subject) | `testSuite` |
-
 ### `artifact`
 
 An `artifact` is usually produced as output of a build process. Events need to be generated to indicate that an `artifact` has been packaged and released for others to use. These events can be produced by the artifact producer or by the artifact storage system.
@@ -116,76 +92,6 @@ This event represents a Build task that has finished. This event will eventually
 | type | `String` | See [type](spec.md#type-subject) | |
 | artifactId | `Purl` | Identifier of the artifact produced by the build | `pkg:oci/myapp@sha256%3A0b31b1c02ff458ad9b7b81cbdf8f028bd54699fa151f221d1e8de6817db93427`, `pkg:golang/mygit.com/myorg/myapp@234fd47e07d1004f0aed9c` | `build` | |
 
-### `testCase queued`
-
-This event represents a Test task that has been queued, and it is waiting to be started.
-
-- Event Type: __`dev.cdevents.testcase.queued.0.1.1`__
-- Predicate: queued
-- Subject: [`testCase`](#testcase)
-
-| Field | Type | Description | Examples | Required |
-|-------|------|-------------|----------|----------------------------|
-| id    | `String` | See [id](spec.md#id-subject)| `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ |
-| source | `URI-Reference` | See [source](spec.md#source-subject) | | |
-| type | `String` | See [type](spec.md#type-subject) | `testCase` | |
-
-### `testCase started`
-
-This event represents a Test task that has started.
-
-- Event Type: __`dev.cdevents.testcase.started.0.1.1`__
-- Predicate: started
-- Subject: [`testCase`](#testcase)
-
-| Field | Type | Description | Examples | Required |
-|-------|------|-------------|----------|----------------------------|
-| id    | `String` | See [id](spec.md#id-subject)| `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ |
-| source | `URI-Reference` | See [source](spec.md#source-subject) | | |
-| type | `String` | See [type](spec.md#type-subject) | `testCase` | |
-
-### `testCase finished`
-
-This event represents a Test task that has finished. This event will eventually contain the finished status: success, error or failure.
-
-- Event Type: __`dev.cdevents.testcase.finished.0.1.1`__
-- Predicate: finished
-- Subject: [`testCase`](#testcase)
-
-| Field | Type | Description | Examples | Required |
-|-------|------|-------------|----------|----------------------------|
-| id    | `String` | See [id](spec.md#id-subject)| `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ |
-| source | `URI-Reference` | See [source](spec.md#source-subject) | | |
-| type | `String` | See [type](spec.md#type-subject) | `testCase` | |
-
-### `testSuite started`
-
-This event represents a Test suite that has been started.
-
-- Event Type: __`dev.cdevents.testsuite.started.0.1.1`__
-- Predicate: started
-- Subject: [`testSuite`](#testsuite)
-
-| Field | Type | Description | Examples | Required |
-|-------|------|-------------|----------|----------------------------|
-| id    | `String` | See [id](spec.md#id-subject)| `unit`, `e2e`, `security` | ✅ |
-| source | `URI-Reference` | See [source](spec.md#source-subject) | | |
-| type | `String` | See [type](spec.md#type-subject) | `testSuite` | |
-
-### `testSuite finished`
-
-This event represents a Test suite that has has finished, the event will contain the finished status: success, error or failure.
-
-- Event Type: __`dev.cdevents.testsuite.finished.0.1.1`__
-- Predicate: finished
-- Subject: [`testSuite`](#testsuite)
-
-| Field | Type | Description | Examples | Required |
-|-------|------|-------------|----------|----------------------------|
-| id    | `String` | See [id](spec.md#id-subject)| `unit`, `e2e`, `security` | ✅ |
-| source | `URI-Reference` | See [source](spec.md#source-subject) | | |
-| type | `String` | See [type](spec.md#type-subject) | `testSuite` | |
-
 ### `artifact packaged`
 
 The event represents an artifact that has been packaged for distribution; this artifact is now versioned with a fixed version.

examples/testcaserun_finished.json

@@ -0,0 +1,27 @@
+{
+  "context": {
+    "version": "0.3.0-draft",
+    "id": "271069a8-fc18-44f1-b38f-9d70a1695819",
+    "source": "/event/source/123",
+    "type": "dev.cdevents.testcaserun.finished.0.1.0-draft",
+    "timestamp": "2023-03-20T14:27:05.315384Z"
+  },
+  "subject": {
+    "id": "myTestCaseRun123",
+    "source": "/event/source/123",
+    "type": "testCaseRun",
+    "content": {
+      "outcome" : "pass",
+      "environment": {
+        "id": "dev",
+        "source": "testkube-dev-123"
+      },
+      "testCase": {
+        "id": "92834723894",
+        "version": "1.0",
+        "name": "Login Test",
+        "type": "integration"
+      }
+    }
+  }
+}

examples/testcaserun_queued.json

@@ -0,0 +1,29 @@
+{
+  "context": {
+    "version": "0.3.0-draft",
+    "id": "271069a8-fc18-44f1-b38f-9d70a1695819",
+    "source": "/event/source/123",
+    "type": "dev.cdevents.testcaserun.queued.0.1.0-draft",
+    "timestamp": "2023-03-20T14:27:05.315384Z"
+  },
+  "subject": {
+    "id": "myTestCaseRun123",
+    "source": "/event/source/123",
+    "type": "testCaseRun",
+    "content": {
+      "environment": {
+        "id": "dev",
+        "source": "testkube-dev-123"
+      },
+      "testCase": {
+        "id": "92834723894",
+        "version": "1.0",
+        "name": "Login Test",
+        "type": "integration"
+      },
+      "trigger": {
+        "type": "schedule"
+      }
+    }
+  }
+}

examples/testcaserun_started.json

@@ -0,0 +1,29 @@
+{
+  "context": {
+    "version": "0.3.0-draft",
+    "id": "271069a8-fc18-44f1-b38f-9d70a1695819",
+    "source": "/event/source/123",
+    "type": "dev.cdevents.testcaserun.started.0.1.0-draft",
+    "timestamp": "2023-03-20T14:27:05.315384Z"
+  },
+  "subject": {
+    "id": "myTestCaseRun123",
+    "source": "/event/source/123",
+    "type": "testCaseRun",
+    "content": {
+      "environment": {
+        "id": "dev",
+        "source": "testkube-dev-123"
+      },
+      "testCase": {
+        "id": "92834723894",
+        "version": "1.0",
+        "name": "Login Test",
+        "type": "integration"
+      },
+      "trigger": {
+        "type": "schedule"
+      }
+    }
+  }
+}

examples/testoutput_published.json

@@ -0,0 +1,22 @@
+{
+  "context": {
+    "version": "0.3.0-draft",
+    "id": "271069a8-fc18-44f1-b38f-9d70a1695819",
+    "source": "/event/source/123",
+    "type": "dev.cdevents.testoutput.published.0.1.0-draft",
+    "timestamp": "2023-03-20T14:27:05.315384Z"
+  },
+  "subject": {
+    "id": "testrunreport-12123",
+    "source": "/event/source/testrunreport-12123",
+    "type": "testOutput",
+    "content": {
+      "outputType" : "video",
+      "format" : "video/quicktime",
+      "testCaseRun": {
+        "id": "myTestCaseRun123",
+        "source": "testkube-dev-123"
+      }
+    }
+  }
+}

examples/testsuiterun_finished.json

@@ -0,0 +1,28 @@
+{
+  "context": {
+    "version": "0.3.0-draft",
+    "id": "271069a8-fc18-44f1-b38f-9d70a1695819",
+    "source": "/event/source/123",
+    "type": "dev.cdevents.testsuiterun.finished.0.1.0-draft",
+    "timestamp": "2023-03-20T14:27:05.315384Z"
+  },
+  "subject": {
+    "id": "myTestSuiteRun123",
+    "source": "/event/source/123",
+    "type": "testSuiteRun",
+    "content": {
+      "outcome" : "fail",
+      "severity" : "critical",
+      "reason" : "Host 123.34.23.32 not found",
+      "environment": {
+        "id": "dev",
+        "source": "testkube-dev-123"
+      },
+      "testSuite": {
+        "id": "92834723894",
+        "version": "1.0",
+        "name": "Auth TestSuite"
+      }
+    }
+  }
+}

examples/testsuiterun_queued.json

@@ -0,0 +1,28 @@
+{
+  "context": {
+    "version": "0.3.0-draft",
+    "id": "271069a8-fc18-44f1-b38f-9d70a1695819",
+    "source": "/event/source/123",
+    "type": "dev.cdevents.testsuiterun.queued.0.1.0-draft",
+    "timestamp": "2023-03-20T14:27:05.315384Z"
+  },
+  "subject": {
+    "id": "myTestSuiteRun123",
+    "source": "/event/source/123",
+    "type": "testSuiteRun",
+    "content": {
+      "environment": {
+        "id": "dev",
+        "source": "testkube-dev-123"
+      },
+      "testSuite": {
+        "id": "92834723894",
+        "version": "1.0",
+        "name": "Auth TestSuite"
+      },
+      "trigger": {
+        "type": "pipeline"
+      }
+    }
+  }
+}