feat: use labels for test execution on PRs to main (#208)

## Context So far, PRs to `main` by default have all the tests enabled. Applying labels to enable particular tests is available only for PRs that are not to `main`. Our team doesn't use `develop` branch, thus we need to be able to skip some tests on PRs to `main` - to get feedback quickly. ## Summary of changes - This PR makes it possible to use labels for enabling/disabling tests also on PRs to `main`. - apart from test labels (`execute_knowledge`, etc.), `use_labels` label has to be applied to the PR before (re)opening it to make test labels work - this is to ensure that execution of all tests is still the default option for PRs to `main` ## Sample CI/CD runs for PR PR: splunk/splunk-add-on-for-microsoft-cloud-services#993 Workflow runs: - with `use_labels` label: https://github.com/splunk/splunk-add-on-for-microsoft-cloud-services/actions/runs/7338668760 - with `use_labels` and `execute_knowledge` labels: https://github.com/splunk/splunk-add-on-for-microsoft-cloud-services/actions/runs/7338999846/job/19982527979 - with no labels: https://github.com/splunk/splunk-add-on-for-microsoft-cloud-services/actions/runs/7339852310/job/19984825125 --------- Co-authored-by: Artem Rys <[email protected]>
splunk · Dec 29, 2023 · 3af5fd6 · 3af5fd6
1 parent 4f3fa4d
commit 3af5fd6
Show file tree

Hide file tree

Showing 2 changed files with 31 additions and 77 deletions.
diff --git a/.github/workflows/reusable-build-test-release.yml b/.github/workflows/reusable-build-test-release.yml
@@ -157,12 +157,18 @@ jobs:
 
           case "${{ github.event_name }}" in
           "pull_request")
-            if ${{ github.base_ref == 'main' }} || ${{ contains(github.event.pull_request.labels.*.name, 'execute_all_tests') }}; then
+            labels=$(echo '${{ toJSON(github.event.pull_request.labels) }}' | jq -r '.[] | .name')
+            if ${{ github.base_ref == 'main' }} && ${{ contains(github.event.pull_request.labels.*.name, 'use_labels') }}; then
+              for test_type in "${TESTSET[@]}"; do
+                if [[ "$labels" =~ $test_type ]]; then
+                  EXECUTE_LABELED["$test_type"]="true"
+                fi
+              done
+            elif ${{ github.base_ref == 'main' }} || ${{ contains(github.event.pull_request.labels.*.name, 'execute_all_tests') }}; then
               for test_type in "${TESTSET[@]}"; do
                 EXECUTE_LABELED["$test_type"]="true"
               done
             else
-              labels=$(echo '${{ toJSON(github.event.pull_request.labels) }}' | jq -r '.[] | .name')
               for test_type in "${TESTSET[@]}"; do
                 if [[ "$labels" =~ $test_type ]]; then
                   EXECUTE_LABELED["$test_type"]="true"
@@ -2699,7 +2705,7 @@ jobs:
       - name: exit without publish
         if: ${{ steps.check.outputs.run-publish == 'false' || ( github.event.action == 'labeled' && github.event.label.name == 'preserve_infra' ) }}
         run: |
-          echo " Some test job failed or Workflow has triggred on preserve_infra label. "
+          echo "Some test job failed or Workflow has triggered on preserve_infra label."
           exit 1
 
   publish:

diff --git a/README.md b/README.md
@@ -1,86 +1,34 @@
-# addonfactory-workflow-addon-release
-Repository to store reusable `build-test-release` workflow, which is used to release Splunk add-ons. 
-Worklow is used by add-ons created and managed by [addonfactory repository template](https://github.com/splunk/addonfactory-repository-template)
-Workflow defines jobs which perform security code scanning, execute different types of tests, build add-on package, make github release.
-
-## Example usage
-```yaml
-name: build-test-release
-on:
-  push:
-    branches:
-      - "main"
-    tags:
-      - "v[0-9]+.[0-9]+.[0-9]+"
-  pull_request:
-    branches: 
-      - "**"
-# explicitly configure permissions, in case your GITHUB_TOKEN workflow permissions are set to read-only in repository settings
-permissions:
-  actions: read
-  checks: write
-  contents: write
-  deployments: read
-  packages: write
-  pull-requests: read
-  statuses: write
-jobs:
-  call-workflow:
-    uses: splunk/addonfactory-workflow-addon-release/.github/workflows/[email protected]
-    secrets:
-      GH_TOKEN_ADMIN: ${{ secrets.GH_TOKEN_ADMIN }}
-      SEMGREP_PUBLISH_TOKEN: ${{ secrets.SEMGREP_KEY }}
-      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
-      AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
-      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
-      VT_API_KEY: ${{ secrets.VT_API_KEY }}
-      OTHER_TA_REQUIRED_CONFIGS: ${{ secrets.OTHER_TA_REQUIRED_CONFIGS }}
-      FOSSA_API_KEY: ${{ secrets.FOSSA_API_KEY }}
-      SA_GH_USER_NAME: ${{ secrets.SA_GH_USER_NAME }}
-      SA_GH_USER_EMAIL: ${{ secrets.SA_GH_USER_EMAIL }}
-      SA_GPG_PRIVATE_KEY: ${{ secrets.SA_GPG_PRIVATE_KEY }}
-      SA_GPG_PASSPHRASE: ${{ secrets.SA_GPG_PASSPHRASE }}
-      SPL_COM_USER: ${{ secrets.SPL_COM_USER }}
-      SPL_COM_PASSWORD: ${{ secrets.SPL_COM_PASSWORD }}
-```
-
-***
+# Reusable workflow
 
-# Troubleshooting for different workflow stages in Github Actions
-
-General troubleshooting 
-=======================
+This repository stores reusable `build-test-release` workflow, which is used to build, test and release Splunk add-ons.
 
-- For each stage there are logs which provides list of failures or link the test report for the stage or more details like error code regarding what caused the stage to fail.
+Workflow is used by add-ons created and managed by [addonfactory repository template](https://github.com/splunk/addonfactory-repository-template).
 
-- If execution of stage takes longer than expected and we can re-trigger a workflow with an empty commit to check is the issue with the stage is consistent.
+Workflow defines jobs which perform security code scanning, execute different types of tests, build add-on package and make a GitHub release.
 
-- If the stage starts failing without any code change we can check the version of action used in the Github actions pipeline and check the behaviour with previous releases.
+# Troubleshooting for different workflow stages in GitHub Actions
 
-- Check if there is any similar issue reported to Github repo for the action by other users.
+## General troubleshooting
 
-- Validate If the failure is caused by a code change in the action which modified behaviour in the latest release causing the stage to fail.
+* For each stage there are logs which provides list of failures or link the test report for the stage or more details like error code regarding what caused the stage to fail.
+* Check if there is any similar issue reported to GitHub repo for the action by other users.
+* If you are not sure what to do, please use `go/addon/help`.
 
+## setup-workflow
 
-setup-workflow
-=======================
-
-**Description:**
-- Job that is scanning pull_request and based on PR body or included labels defining tests to be executed or infrastructures to be preserved. 
-  - To preserve infrastructure:
-    - add to PR label `preserve_infra`
-    - add to PR description add `preserve: {comma separated list of test type}`
-    - available choices: `knowledge ui modinput_functional scripted_inputs escu requirement_test`
-    - to trigger tests again, reapply `preserve_infra` label
-  - To trigger specified test type
-    - add to PR one or multiple labels
-    - available choices: `execute_knowledge execute_ui execute_modinput_functional execute_scripted_inputs execute_escu execute_requirement_test execute_all_tests`
-    - adding labels will result retriggering job
-  - All tests are executed by default when:
-    - PR target branch is 'main'
-    - PUSH event on branches 'main', 'develop' and on tags (on release)
-    - SCHEDULE event
+Job that is scanning PR and based on PR body or included labels defining tests to be executed or infrastructures to be preserved.
 
+* To preserve infrastructure
+  * add to PR label `preserve_infra`
+  * add to PR description add `preserve: {comma separated list of test type}`, available choices: `knowledge ui modinput_functional scripted_inputs requirement_test`
+  * to trigger tests again, reapply `preserve_infra` label
+* All tests are executed by default when (controlled from [here](https://github.com/splunk/addonfactory-repository-template/blob/main/enforce/.github/workflows/build-test-release.yml))
+  * PR target branch is `main` (unless `use_labels` label is used then specific test labels (see below) should be added to execute specific test types) 
+  * push event on branches `main`, `develop` and on `tags` (on release)
+  * schedule event (controlled from [here](https://github.com/splunk/addonfactory-repository-template/blob/main/tools/jinja_parameters.yml))
+* To trigger specific test type
+  * add to PR one or multiple labels, available choices can be found [here](https://github.com/splunk/addonfactory-workflow-addon-release/blob/4f3fa4d779b6ec7649f0dc6b973eb4d68e5fcc48/.github/workflows/reusable-build-test-release.yml#L153)
+  * there is no need to add labels when PR's target branch is `main`
 
 meta stage
 =======================