Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: make collectors repository configurable #739

Draft
wants to merge 1 commit into
base: development
Choose a base branch
from
Draft

feat: make collectors repository configurable #739

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 13 additions & 9 deletions pipelines/run-collectors/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,18 @@ save the required resources to the workspace, execute the collectors, then updat

## Parameters

| Name | Description | Optional | Default value |
|------------------------|-------------------------------------------------------------------------------------------------------------------|----------|-----------------------------------------------------------|
| release | The namespaced name (namespace/name) of the Release custom resource initiating this pipeline execution | No | - |
| previousRelease | The namespaced name (namespace/name) of the previous successful Release prior to the one passed as params.release | No | - |
| collectorsResourceType | The type of resource that contains the collectors | No | - |
| collectorsResource | The namespaced name of the resource that contains the collectors | No | - |
| taskGitUrl | The url to the git repo where the release-service-catalog tasks to be used are stored | Yes | https://github.com/konflux-ci/release-service-catalog.git |
| taskGitRevision | The revision in the taskGitUrl repo to be used | No | - |
| Name | Description | Optional | Default value |
|------------------------|-------------------------------------------------------------------------------------------------------------------|----------|--------------------------------------------------------------|
| release | The namespaced name (namespace/name) of the Release custom resource initiating this pipeline execution | No | - |
| previousRelease | The namespaced name (namespace/name) of the previous successful Release prior to the one passed as params.release | No | - |
| collectorsResourceType | The type of resource that contains the collectors | No | - |
| collectorsResource | The namespaced name of the resource that contains the collectors | No | - |
| collectorsRepository | Git repository where the collectors will be defined | Yes | https://github.com/konflux-ci/release-service-collectors.git |
| taskGitUrl | The url to the git repo where the release-service-catalog tasks to be used are stored | Yes | https://github.com/konflux-ci/release-service-catalog.git |
| taskGitRevision | The revision in the taskGitUrl repo to be used | No | - |

## Changes in 0.3.0
* Add a new parameter to define the collectors repository

## Changes in 0.2.0
* Stop passing releasePlan and releasePlanAdmission and pass instead a reference to a resource
* Stop passing releasePlan and releasePlanAdmission and pass instead a reference to a resource
8 changes: 7 additions & 1 deletion pipelines/run-collectors/run-collectors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ kind: Pipeline
metadata:
name: run-collectors
labels:
app.kubernetes.io/version: "0.2.0"
app.kubernetes.io/version: "0.3.0"
annotations:
tekton.dev/pipelines.minVersion: "0.12.1"
tekton.dev/tags: release
Expand All @@ -31,6 +31,10 @@ spec:
description: |
The namespaced name of the resource that contains the collectors
type: string
- name: collectorsRepository
type: string
description: Git repository where the collectors will be defined
default: https://github.com/konflux-ci/release-service-collectors.git
- name: taskGitUrl
type: string
description: The url to the git repo where the release-service-catalog tasks to be used are stored
Expand Down Expand Up @@ -80,6 +84,8 @@ spec:
value: $(tasks.collect-collectors-resources.results.collectorsResource)
- name: collectorsResourceType
value: $(params.collectorsResourceType)
- name: collectorsRepository
value: $(params.collectorsRepository)
- name: resultsDir
value: $(tasks.collect-collectors-resources.results.resultsDir)
workspaces:
Expand Down
6 changes: 6 additions & 0 deletions tasks/run-collectors/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ one file per collector.
| collectorsPath | Path to the JSON string of the resource containing the collectors in the data workspace | No | - |
| collectorsResourceType | The type of resource that contains the collectors | No | - |
| resultsDir | The relative path in the workspace to save the collector results to | No | - |
| collectorsRepository | Git repository where the collectors will be defined | No | - |

## Changes in 0.3.0
* Add a new parameter to define the collectors repository
* Force a cd into /tmp to avoid errors while cloning
* Address the change in the collectors spec

## Changes in 0.2.0
* Pass reference to the collectors path and to the resource type
11 changes: 8 additions & 3 deletions tasks/run-collectors/run-collectors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ kind: Task
metadata:
name: run-collectors
labels:
app.kubernetes.io/version: "0.2.0"
app.kubernetes.io/version: "0.3.0"
annotations:
tekton.dev/pipelines.minVersion: "0.12.1"
tekton.dev/tags: release
Expand All @@ -21,6 +21,9 @@ spec:
- name: resultsDir
type: string
description: The relative path in the workspace to save the collector results to
- name: collectorsRepository
type: string
description: Git repository where the collectors will be defined
workspaces:
- name: data
description: Workspace where the CRs are stored
Expand All @@ -32,7 +35,7 @@ spec:
set -xeo pipefail

DEFAULT_TIMEOUT=900
COLLECTORS_REPOSITORY="https://github.com/SOME_ORG/SOME_REPO" # TODO - update this once repo is created
COLLECTORS_REPOSITORY="$(params.collectorsRepository)"
COLLECTORS_FILE="$(workspaces.data.path)/$(params.collectorsPath)"
COLLECTORS_RESOURCE_TYPE="$(params.collectorsResourceType)"
RESULTS_DIR="$(workspaces.data.path)/$(params.resultsDir)"
Expand Down Expand Up @@ -60,6 +63,8 @@ spec:
exit 1
fi

cd /tmp

git clone "${COLLECTORS_REPOSITORY}" collectors
pushd collectors

Expand All @@ -80,7 +85,7 @@ spec:
else
NUM_COLLECTORS="$(jq '.spec.collectors | length' "${COLLECTORS_FILE}")"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aren't you missing items here?

for ((i = 0; i < NUM_COLLECTORS; i++)) ; do
collector="$(jq -c --argjson i "$i" '.spec.collectors[$i]' "${COLLECTORS_FILE}")"
collector="$(jq -c --argjson i "$i" '.spec.collectors.items[$i]' "${COLLECTORS_FILE}")"
# Limit batch size to concurrent limit
while (( ${RUNNING_JOBS@P} >= "$CONCURRENT_LIMIT" )); do
wait -n
Expand Down
24 changes: 13 additions & 11 deletions tasks/run-collectors/tests/test-run-collectors-fail-timeout.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,19 @@ spec:
},
"spec": {
"application": "app",
"collectors": [
{
"name": "test-collector",
"type": "timeout-collector",
"timeout": 1,
"params": [
{
"name": "test-arg",
"value": "test-value"
}
]
"collectors":
"items": [
{
"name": "test-collector",
"type": "timeout-collector",
"timeout": 1,
"params": [
{
"name": "test-arg",
"value": "test-value"
}
]
}
}
],
"target": "managed"
Expand Down
24 changes: 13 additions & 11 deletions tasks/run-collectors/tests/test-run-collectors-parallel.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,17 +31,19 @@ spec:
},
"spec": {
"application": "app",
"collectors": [
{
"name": "test-collector",
"type": "parallel-collector",
"timeout": 600,
"params": [
{
"name": "test-arg",
"value": "test-value"
}
]
"collectors":
"items": [
{
"name": "test-collector",
"type": "parallel-collector",
"timeout": 600,
"params": [
{
"name": "test-arg",
"value": "test-value"
}
]
}
}
],
"target": "managed"
Expand Down
24 changes: 13 additions & 11 deletions tasks/run-collectors/tests/test-run-collectors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,17 +30,19 @@ spec:
},
"spec": {
"application": "app",
"collectors": [
{
"name": "test-collector",
"type": "dummy-collector",
"timeout": 600,
"params": [
{
"name": "test-arg",
"value": "test-value"
}
]
"collectors":
"items": [
{
"name": "test-collector",
"type": "dummy-collector",
"timeout": 600,
"params": [
{
"name": "test-arg",
"value": "test-value"
}
]
}
}
],
"target": "managed"
Expand Down
Loading