Skip to content

Latest commit

 

History

History
224 lines (159 loc) · 9.61 KB

manage-component-jobs-with-templates.md

File metadata and controls

224 lines (159 loc) · 9.61 KB
Raw

Manage component jobs with templates

This document describes how to define, modify, and remove Prow jobs for Kyma components using predefined templates that create both presubmit and postsubmit jobs for your component. Also, this document gives you the steps required to prepare your component for the Prow CI pipeline.

NOTE: There are two templates that you can use to generate your component job definitions, component.yaml and generic-component.yaml. Use the recommended generic-component.yaml template that is the latest version using a generic bootstrap. If you decide to use component.yaml, define a proper buildpack.

Add component jobs

Follow these steps:

  1. Edit the configuration file.

Go to templates/config.yaml and add a new entry with your component details to the render list under the templates section.

See an example that defines the application-broker component from the kyma repository, using the generic bootstrap:

  - from: templates/generic-component.yaml
    render:
      - to: ../prow/jobs/kyma/components/application-broker/application-broker-generic.yaml
        values:
          <<: *kyma_generic_component
          path: components/application-broker
          since: "1.7"
          optional: true
    ...

Such an entry uses the generic-component.yaml template to create the application-broker-generic.yaml file under the /prow/jobs/kyma/components/application-broker/ subfolder, specifying that the presubmit and postsubmit jobs for this component should apply from the 1.7 release onwards. Set the optional parameter to true for this job to be optional on pull requests (PRs), not to block others.

NOTE: Make sure that the .yaml file and the component folder name are the same as the name of the Kyma component. Also, all .yaml files in the whole jobs structure need to have unique names.

Use the buildpack for Go or Node.js applications provided in the test-infra repository. It is the standard mechanism for defining Prow jobs. If the buildpack you want to use is not there yet, you have to add it. When you add a new buildpack, follow the example of the already defined ones.

  1. Define a test for your component.

Add a new component test entry to the components_test.go file for the test-infra-test-jobs-yaml-definitions presubmit job to execute it.

See the example:

...
{path: "apiserver-proxy", image: tester.ImageGolangBuildpack1_12},
{path: "apiserver-proxy", image: tester.ImageBootstrap20181204, suite: tester.NewGenericComponentSuite,
  additionalOptions: []jobsuite.Option{
    jobsuite.JobFileSuffix("generic"),
    jobsuite.Since(releases.Release17),
    jobsuite.Optional(),
  },
},

Same as with component jobs, mark the component test as optional at this stage by adding the jobsuite.Optional() entry.

If you have access to a Prow cluster, you can test a Prow job on it. For details, see the official documentation.

When writing tests for a new component, use the tester.GetKymaReleasesSince({next release}) function to create tests for release jobs.

  1. Generate jobs.

Run this command to generate jobs previously defined in the config.yaml file:

go run development/tools/cmd/rendertemplates/main.go --config templates/config.yaml

As a result, the Render Templates tool generates the requested job files.

  1. Check your configuration locally.

Use the development/validate-config.sh script to validate your Prow configuration. The script accepts three arguments:

  • The path to the plugins configuration file (prow/plugins.yaml)
  • The path to the generic configuration file (prow/config.yaml)
  • The path to the directory with job definitions (prow/jobs/)

See an example:

cd $GOPATH/src/github.com/kyma-project/test-infra
./development/validate-config.sh prow/plugins.yaml prow/config.yaml prow/jobs/
  1. Merge the changes.

Create a PR with your changes in the config.yaml file and the job files generated by the Render Templates.

After your PR is reviewed and approved, merge the changes to the test-infra repository. The job configuration is automatically applied to the Prow production cluster. The config_updater plugin configured in the prow/plugins.yaml file adds a comment to the PR:

msg.

  1. Create a Makefile for your component.

Buildpacks require Makefile defined in your component directory under the kyma repository. The Makefile has to define the ci-release target that is executed for a PR issued against the release branch.

See an example of Makefile for the Console Backend Service component that already uses the generic buildpack:

APP_NAME = console-backend-service
APP_PATH = components/$(APP_NAME)
BUILDPACK = eu.gcr.io/kyma-project/test-infra/buildpack-golang-toolbox:v20190930-d28d219
SCRIPTS_DIR = $(realpath $(shell pwd)/../..)/scripts

include $(SCRIPTS_DIR)/go-dep.mk

VERIFY_IGNORE := /vendor\|/automock\|/testdata\|/pkg

.PHONY: path-to-referenced-charts
path-to-referenced-charts:
	@echo "resources/core"

NOTE Add a tab before each command.

If your job involves pushing a Docker image, its name is based on the following environment variables:

  • DOCKER_TAG that refers to the Docker tag set by the build.sh script.
  • DOCKER_PUSH_DIRECTORY that points to the directory in the Docker repository where the image is pushed. Set it in the job definition by adding the preset-build-pr, preset-build-master, or preset-build-release Preset.
  • DOCKER_PUSH_REPOSITORY that is the Docker repository where the image is pushed. It is set in the job definition by the preset-docker-push-repository Preset.
  1. Change your component job and test to obligatory.

Create another PR in the test-infra repository that removes these entries:

  • optional: true from your component job definition in templates/config.yaml.
  • jobsuite.Optional() from your component test definition in components_test.go.

This change makes your component job and test obligatory to pass on all PRs before they can be merged.

Modify component jobs

To change component job configuration, follow these steps:

  1. In the config.yaml file, change the name of the file where the jobs are generated. For example, add the deprecated suffix.

  2. Add until: {last release} to this configuration. It specifies the release until which this component version applies.

  3. Create a new entry with the new configuration. Set the to field to point to the file responsible for storing jobs.

  4. Add since: {next release} to the new entry. It specifies the release from which this component version applies.

    See this example:

    Buildpack for the API Controller changed from go1.11 to go.12 in release 1.5. This is the component configuration before the buildpack change:

       - to: ../prow/jobs/kyma/components/api-controller/api-controller.yaml
     values:
       <<: *go_kyma_component_1_11
       path: components/api-controller

    This is what the configuration created after the buildpack change looks like:

       - to: ../prow/jobs/kyma/components/api-controller/api-controller.yaml
     values:
       <<: *go_kyma_component_1_12
       path: components/api-controller
       since: '1.5'
   - to: ../prow/jobs/kyma/components/api-controller/api-controller-deprecated.yaml
     values:
       <<: *go_kyma_component_1_11
       path: components/api-controller
       until: '1.4'

  5. Modify tests.

    Add a new entry to component tests and modify the existing one to specify the release version until which the tests apply.

    See the example of the Console Backend Service:

    ...
{path: "console-backend-service", image: tester.ImageGolangBuildpack1_11,
  additionalOptions: []jobsuite.Option{
    jobsuite.Until(releases.Release15),
  },
},
{path: "console-backend-service", image: tester.ImageBootstrap20181204, suite: tester.NewGenericComponentSuite,
  additionalOptions: []jobsuite.Option{
    jobsuite.JobFileSuffix("generic"),
    jobsuite.Since(releases.Release16),
    jobsuite.RunIfChanged("components/console-backend-service/main.go", "scripts/go-dep.mk"),
  },
},

    When changing tests, use the tester.GetKymaReleasesUntil({last release}) function in place of tester.GetAllKymaReleases to test older releases. Use the tester.GetKymaReleasesSince({next release}) function to create tests for release jobs for future releases.

Remove component jobs

CI pipeline in Kyma supports jobs for three last releases so plan the component job removal in advance. Before you remove your component from Prow, add the until: '{release}' entry to your component job definition in the templates/config.yaml file.

For example, if you are planning to remove your component after version 1.3, add the until: '1.3' entry to your component job definition and remove it only when the release 1.3 is no longer supported:

global:
  nextRelease: "1.7"
  releases:
    - "1.6"
    - "1.5"
    - "1.4"
...

To remove a component from Prow, follow these steps:

  1. In the config.yaml file, remove the entries under the templates section that refer to your component.
  2. Manually remove all files and the component folder from /prow/jobs.
  3. Delete tests for the component jobs.