DOCUMENT.md

Testing Document

This document describes how to define your tests with YAML. For the plugin cli usage, please check README

• Test Suite
• Test Job
• Assertion
  • Assertion Types
  • Antonym and not

Test Suite

A test suite is a collection of tests with the same purpose and scope defined in one single file. The root structure of a test suite is like below:

suite: test deploy and service
templates:
  - templates/deployment.yaml
  - templates/web/service.yaml
release:
  name: my-release
  namespace: my-namespace
  revision: 1
  upgrade: true
capabilities:
  majorVersion: 1
  minorVersion: 10
  apiVersions:
    - br.dev.local/v2
chart:
  version: 1.0.0
  appVersion: 1.0.0
tests:
  - it: should test something
    ...

• suite: string, optional. The suite name to show on test result output.

• templates: array of string, recommended. The template files scope to test in this suite. The full chart will be rendered, however only the listed templates are filtered for validation. Template files that are put in a templates sub-folder can be addressed with a linux path separator. Also the templates/ can be omitted. Partial templates (which are prefixed with and _) are added automatically even if it is in a templates sub-folder, you don't need to add them again.

• release: object, optional. Define the {{ .Release }} object.

  • name: string, optional. The release name, default to "RELEASE-NAME".
  • namespace: string, optional. The namespace which release be installed to, default to "NAMESPACE".
  • revision: string, optional. The revision of current build, default to 0.
  • upgrade: bool, optional. Whether the build is an upgrade, default to false.

• capabilities: object, optional. Define the {{ .Capabilities }} object.

  • majorVersion: string, optional. The kubernetes major version, default to the major version which is set by helm.
  • minorVersion: string, optional. The kubernetes minor version, default to the minor version which is set by helm.
  • apiVersions: array of string, optional. A set of versions, default to the versionset used by the defined kubernetes version.

• chart: object, optional. Define the {{ .Chart }} object.

  • version: string, optional. The semantic version of the chart, default to the version set in the Chart.
  • appVersion: string, optional. The app-version of the chart, default to the app-version set in the Chart.

• tests: array of test job, required. Where you define your test jobs to run, check Test Job.

Test Job

The test job is the base unit of testing. Your chart is rendered each time a test job run, and validated with assertions defined in the test. You can setup your values used to render the chart in the test job with external values files or directly in the test job definition. Below is a test job example with all of its options defined:

...
tests:
  - it: should pass
    values:
      - ./values/staging.yaml
    set:
      image.pullPolicy: Always
      resources:
        limits:
          memory: 128Mi
    template: deployment.yaml
    documentIndex: 0
    release:
      name: my-release
      namespace:
      revision: 9
      upgrade: true
    capabilities:
      majorVersion: 1
      minorVersion: 12
      apiVersions:
        - custom.api/v1
    chart:
      version: 1.0.0
      appVersion: 1.0.0
    asserts:
      - equal:
          path: metadata.name
          value: my-deploy

• it: string, recommended. Define the name of the test with TDD style or any message you like.

• values: array of string, optional. The values files used to renders the chart, think it as the -f, --values options of helm install. The file path should be the relative path from the test suite file itself.

• set: object of any, optional. Set the values directly in suite file. The key is the value path with the format just like --set option of helm install, for example image.pullPolicy. The value is anything you want to set to the path specified by the key, which can be even an array or an object.

• template: string, optional. templates: array of string, optional. The template file(s) which render the manifest to be tested, default to the list of template file defined in templates of suite file, unless template is defined in the assertion(s) (check Assertion).

• documentIndex: int, optional. The index of rendered documents (divided by ---) to be tested, default to -1, which results in asserting all documents (see Assertion). Generally you can ignored this field if the template file render only one document.

• release: object, optional. Define the {{ .Release }} object.

  • name: string, optional. The release name, default to "RELEASE-NAME".
  • namespace: string, optional. The namespace which release be installed to, default to "NAMESPACE".
  • revision: string, optional. The revision of current build, default to 0.
  • upgrade: bool, optional. Whether the build is an upgrade, default to false.

• capabilities: object, optional. Define the {{ .Capabilities }} object.

  • majorVersion: string, optional. The kubernetes major version, default to the major version which is set by helm.
  • minorVersion: string, optional. The kubernetes minor version, default to the minor version which is set by helm.
  • apiVersions: array of string, optional. A set of versions, default to the versionset used by the defined kubernetes version.

• chart: object, optional. Define the {{ .Chart }} object.

  • version: string, optional. The semantic version of the chart, default to the version set in the Chart.
  • appVersion: string, optional. The app-version of the chart, default to the app-version set in the Chart.

• asserts: array of assertion, required. The assertions to validate the rendered chart, check Assertion.

Assertion

Define assertions in the test job to validate the manifests rendered with values provided. The example below tests the instances' name with 2 equal assertion.

templates:
  - deployment.yaml
  - web/service.yaml
tests:
  - it: should pass
    asserts:
      - equal:
          path: metadata.name
          value: my-deploy
      - equal:
          path: metadata.name
          value: your-service
        not: true
        template: web/service.yaml
        documentIndex: 0

The assertion is defined with the assertion type as the key and its parameters as value, there can be only one assertion type key exists in assertion definition object. And there are three more options can be set at root of assertion definition:

• not: bool, optional. Set to true to assert contrarily, default to false. The second assertion in the example above asserts that the service name is NOT your-service.

• template: string, optional. The template file which render the manifest to be asserted, default to the list of template file defined in templates of suite file, unless the template is in the testjob (see TestJob). For example the first assertion above with no template specified asserts for both deployment.yaml and service.yaml by default. If no template file specified in neither suite, testjob and assertion, the assertion returns an error and fail the test.

• documentIndex: int, optional. The index of rendered documents (divided by ---) to be asserted, default to -1, which will assert all documents. Generally you can ignored this field if the template file render only one document.

Map keys in path containing periods (.) are supported with the use of a jq-like syntax:

- equal:
    path: metadata.annotations.[kubernetes.io/ingress.class]
    value: nginx

Assertion Types

Available assertion types are listed below:

Assertion Type	Parameters	Description	Example
containsDocument	kind: string. Expected kind of manifest. apiVersion: string. Expected apiVersion of manifest. name: string, optional. The value of the metadata.name. namespace: string, optional. The value of the metadata.namespace.	Asserts the documents rendered by the kind and apiVersion specified.	containsDocument: kind: Deployment apiVersion: apps/v1 name:foo namespace: bar
contains	path: string. The set path to assert, the value must be an array. content: any. The content to be contained. count: int, optional. The count of content to be contained. any: bool, optional. ignores any other values within the found content.	Assert the array as the value of specified path contains the content.	contains: path: spec.ports content: name: web port: 80 targetPort: 80 protocle:TCP contains: path: spec.ports content: name: web count: 1 any: true
notContains	path: string. The set path to assert, the value must be an array. content: any. The content NOT to be contained.	Assert the array as the value of specified path NOT contains the content.	notContains: path: spec.ports content: name: server port: 80 targetPort: 80 protocle: TCP contains: path: spec.ports content: name: web count: 1 any: true
equal	path: string. The set path to assert. value: any. The expected value.	Assert the value of specified path equal to the value.	equal: path: metadata.name value: my-deploy
notEqual	path: string. The set path to assert. value: any. The value expected not to be.	Assert the value of specified path NOT equal to the value.	notEqual: path: metadata.name value: my-deploy
equalRaw	value: string. Assert the expected value in a NOTES.txt file.	Assert equal to the value.	equalRaw: value: my-deploy
notEqualRaw	value: string. Assert the expected value in a NOTES.txt file not to be.	Assert equal NOT to the value.	notEqual: value: my-deploy
failedTemplate	errorMessage: string. The (human readable) errorMessage that should occur.	Assert the value of errorMessage is the same as the human readable template rendering error. Also allows to match an error that would happen before template execution (ex: validation of values against schema)	failedTemplate: errorMessage: Required value
hasDocuments	count: int. Expected count of documents rendered.	Assert the documents count rendered by the template specified. The documentIndex option is ignored here.	hasDocuments: count: 2
isAPIVersion	of: string. Expected apiVersion of manifest.	Assert the apiVersion value of manifest, is equilevant to: equal: path: apiVersion value: ...	isAPIVersion: of: v2
isEmpty	path: string. The set path to assert.	Assert the value of specified path is empty (null, "", 0, [], {}).	isEmpty: path: spec.tls
isNotEmpty	path: string. The set path to assert.	Assert the value of specified path is NOT empty (null, "", 0, [], {}).	isNotEmpty: path: spec.selector
isKind	of: String. Expected kind of manifest.	Assert the kind value of manifest, is equilevant to: equal: path: kind value: ...	isKind: of: Deployment
isNull	path: string. The set path to assert.	Assert the value of specified path is null.	isNull: path: spec.strategy
isNotNull	path: string. The set path to assert.	Assert the value of specified path is NOT null.	isNotNull: path: spec.replicas
isSubset	path: string. The set path to assert, the value must be an object. content: any. The content to be contained.	Assert the object as the value of specified path that contains the content.	isSubset: path: spec.template content: metadata: labels: app: basic release: MY-RELEASE
isNotSubset	path: string. The set path to assert, the value must be an object. content: any. The content NOT to be contained.	Assert the object as the value of specified path that NOT contains the content.	isSubset: path: spec.template content: metadata: labels: app: basic release: MY-RELEASE
lengthEqual	path: string, optional. The set path to assert the count of array values. paths: string, optional. The set array of paths to assert the count validation of the founded arrays. count: int, optional. The count of the values in the array.	Assert the count of the path or paths.	lengthEqual: path: spec.tls count: 1
matchRegex	path: string. The set path to assert, the value must be a string. pattern: string. The regex pattern to match (without quoting /).	Assert the value of specified path match pattern.	matchRegex: path: metadata.name pattern: -my-chart$
notMatchRegex	path: string. The set path to assert, the value must be a string. pattern: string. The regex pattern NOT to match (without quoting /).	Assert the value of specified path NOT match pattern.	notMatchRegex: path: metadata.name pattern: -my-chat$
matchRegexRaw	pattern: string. The regex pattern to match (without quoting /) in a NOTES.txt file.	Assert the value match pattern.	matchRegexRaw: pattern: -my-notes$
notMatchRegexRaw	pattern: string. The regex pattern NOT to match (without quoting /) in a NOTES.txt file.	Assert the value NOT match pattern.	notMatchRegexRaw: pattern: -my-notes$
matchSnapshot	path: string. The set path for snapshot.	Assert the value of path is the same as snapshotted last time. Check doc below.	matchSnapshot: path: spec
matchSnapshotRaw		Assert the value in the NOTES.txt is the same as snapshotted last time. Check doc below.	matchSnapshotRaw: {}

Antonym and not

Notice that there are some antonym assertions, the following two assertions actually have same effect:

- equal:
    path: kind
    value: Pod
  not: true
# works the same as
- notEqual:
    path: kind
    value: Pod