Document the json test artefacts

validatedpatterns · Oct 10, 2023 · 205e086 · 205e086
1 parent 1390057
commit 205e086
Show file tree

Hide file tree

Showing 2 changed files with 85 additions and 17 deletions.
diff --git a/content/learn/test-artefacts.adoc b/content/learn/test-artefacts.adoc
@@ -0,0 +1,83 @@
+---
+menu:
+  learn:
+    parent: Workflow
+title: Testing Artefacts
+weight: 50
+aliases: /requirements/tested/
+---
+
+:toc:
+
+= Testing Artefacts
+
+In order to be represented in the CI dashboard, testers can create a publically available JSON file (eg. in an AWS bucket) that records the result of the latest test for each combination of pattern, platform, and OpenShift version.
+
+== File Naming
+
+`{pattern}-{platform}-{openshift version}-stable-badge.json`
+
+Example: `medicaldiag-nutanix-4.13-stable-badge.json`
+
+Note: OpenShift verion should be `major.minor` only
+
+== File Contents
+
+[source,json]
+----
+  {
+    "schemaVersion":1,
+    "label":"{text}",
+    
+    /* For now we assume `message` is the same as patternBranch */
+    "message":"{text}",
+    
+    /* passed => green, test failed => red, test setup failed => yellow */
+    "color":"{test result color}",
+    
+    /* eg. x.y.z */
+    "openshiftVersion":"{full openshift version}",
+    
+    /* eg. AWS, GCP, Nutanix, ... */
+    "infraProvider":"{platform}",
+    
+    /* repo name of the pattern, eg. multicloud-gitops */
+    "patternName": "{text}",
+    
+    /* eg. main, stable-N.M */
+    "patternBranch": "{text}",
+    "date":"{YYYY-MM-DD}",
+    
+    /* Who ran the test eg. Red Hat */
+    "testSource": "{company name}",
+    
+    /* eg. Job number */
+    "testID": "{unique id}",
+    
+    /* if publically available */
+    "jenkinsURL":"{path to job}",
+    
+    "debugInfo":"{location of must-gather tarball}",
+  }
+----
+
+=== Example
+
+[source,json]
+----
+  {
+    "schemaVersion":1,
+    "label":"MCG on Nutanix",
+    "message":"main",
+    "color":"green",
+    "openshiftVersion":"4.13.14",
+    "infraProvider":"Nutanix",
+    "patternName": "multicloud-gitops", 
+    "patternBranch": "main", 
+    "date":"2023-10-23",
+    "testSource": "Red Hat"
+    "testID": "13602",
+    "jenkinsURL":"https://jenkins/job/ValidatedPatterns/job/MedicalDiagnosis/job/medicaldiag-gcp-ocp4.13/37/",
+    "debugInfo":"https://storage.cloud.google.com/.../medicaldiag-gcp-ocp4.13.15-13602.tgz",
+  }
+----
diff --git a/content/learn/tested.adoc b/content/learn/tested.adoc
@@ -45,7 +45,7 @@ Qualification is a Validated Patterns TOC decision with input from the pattern o
 
 . Tested Patterns *MUST* have their implementation reviewed by the patterns team to ensure that it is sufficiently flexible to function across a variety of platforms, customer environments, and any relevant verticals.
 . Tested Patterns *MUST* include a standardized architecture drawing, created with (or at least conforming to) the PAC tooling
-. Tested Patterns *MUST* include a written guide for others to follow when demonstrating how the pattern addresses a business problem
+. Tested Patterns *MUST* include a written guide for others to follow when demonstrating the pattern
 . Tested Patterns *MUST* include a test plan covering all features or attributes being highlighted by the demonstration guide.  Negative flow tests (such as resiliency or data retention in the presence of network outages) are also limited to scenarios covered by the demonstration guide.
 +
 The test plan *MUST* define how to validate if the pattern has been successfully deployed and is functionally operational.
@@ -55,22 +55,7 @@ TODO: Convert above link to adoc
 
 . Tested Patterns *MUST* nominate at least one currently supported OpenShift release to test against
 . Tested Patterns *MUST* ensure the test plan passes at least once per quarter
-. Tested Patterns *MUST* create a publically available JSON file (eg. in an AWS bucket) that records the result of the latest test for each combination of pattern, platform, and OpenShift version. Eg.
-[source,json]
-----
-  {
-    "schemaVersion":1,
-    "label":"gcp-4.13",
-    "message":"main",
-    "color":"green",
-    "jenkinsURL":"https://.../",
-    "infraProvider":"gcp",
-    "openshiftVersion":"4.13.14",
-    "debugInfo":"https://...tgz",
-    "date":"2023-10-03"
-  }
-----
-
+. Tested Patterns *MUST* create a publically available JSON file (eg. in an AWS bucket) that records the result of the latest test for each combination of pattern, platform, and OpenShift version. See link:/learn/test-artefacts/[testing artefacts]
 . Tested Patterns *DO NOT* imply an obligation of support for partner or community operators by Red Hat or the pattern owner.
 
 [id="should"]