Skip to content

Latest commit

 

History

History
99 lines (85 loc) · 3.59 KB

quality-metrics.md

File metadata and controls

99 lines (85 loc) · 3.59 KB
Raw

Quality metrics

This document describes reports that provide an overview of the basic quality measures for the Kyma project. See this epic for more information on gathering quality metrics.

Test coverage

Code coverage metrics are gathered directly from component build logs. In the output of almost every job, there is the Total test coverage: xx% string. See the example:

test coverage logs

Since all logs from the build infrastructure are gathered by Stackdriver, the build-in export feature is used to filter out all these lines and send them to BigQuery:

gcp export ui

The following SQL statement aggregates data to the present comparison between last and previous test coverage:

SELECT
     distinct k8s_pod_prow_k8s_io_job
   , FIRST_VALUE(val)
       OVER w1 AS last_result
   , NTH_VALUE(val,2)
       OVER w1 AS previous_result
   , FIRST_VALUE(timestamp)
       OVER w1 AS last_timestamp
   , NTH_VALUE(timestamp, 2)
       OVER w1 AS previous_timestamp
   , format("https://status.build.kyma-project.io/?job=%s", k8s_pod_prow_k8s_io_job)  as link
FROM 
(
  SELECT
      MIN(CAST(REGEXP_EXTRACT(textPayload, r"(\d{1,2})") as int64)) as val
    , labels.k8s_pod_prow_k8s_io_job as k8s_pod_prow_k8s_io_job
    , MAX(timestamp) as timestamp
  FROM `sap-kyma-prow.total_test_coverage.stdout`
  GROUP BY resource.labels.pod_name, labels.k8s_pod_prow_k8s_io_job
) main
where k8s_pod_prow_k8s_io_job not like 'pre%'
  WINDOW w1 AS (
    PARTITION BY k8s_pod_prow_k8s_io_job ORDER BY timestamp DESC
    ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
order by 2 desc

This query is used as the data source for the report created in DataStudio. You can check this report here.

Bugs and regression metrics

GithubStats application is used to gather data on bugs and regressions. It is executed as a Prow job once a day. The data are grabbed by Stackdriver export and forwarded to BigQuery. A JSON object is automatically flattened into multiple columns.

The Prow job creates the following output:

{
  "Issues": {
    "Open": {
      "TotalCount": 387,
      "Bugs": 45,
      "PriorityCritical": 0,
      "Regressions": 0,
      "TestFailing": 6,
      "TestMissing": 11
    },
    "Closed": {
      "TotalCount": 2245,
      "Bugs": 502,
      "PriorityCritical": 128,
      "Regressions": 0,
      "TestFailing": 100,
      "TestMissing": 34
    }
  },
  "Type": "GithubIssuesStatsReport",
  "Owner": "kyma-project",
  "Repository": "kyma",
  "Timestamp": "2019-12-11T06:01:18.770743384Z"
}

The following SQL statement pre-formats the data:

select 
    CAST(jsonPayload.issues.open.totalcount as int64) as open
  , CAST(jsonPayload.issues.closed.totalcount as int64) as closed
  , CAST(jsonPayload.issues.open.bugs as int64) as open_bugs
  , CAST(jsonPayload.issues.open.regressions as int64) as open_regressions
  , CAST((jsonPayload.issues.open.totalcount + jsonPayload.issues.closed.totalcount) as int64) as totalcount
  , jsonPayload.timestamp
  , CONCAT(jsonPayload.owner, "/", jsonPayload.repository) as repository
from `sap-kyma-prow.stats_github.stdout`
order by jsonPayload.timestamp desc

This query is used as a data source for a report created in DataStudio. You can check this report here.