Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KFLUXBUGS-863: introduce results fixes with requisite performance related tuning (dev/stage) #3884

Merged
merged 7 commits into from
Jun 17, 2024
Merged

KFLUXBUGS-863: introduce results fixes with requisite performance related tuning (dev/stage) #3884

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a7a4cae
update components/monitoring/grafana/base/dashboards/pipeline-service…
rh-tap-build-team[bot] Jun 14, 2024
4dc326f
update components/pipeline-service/development/kustomization.yaml
rh-tap-build-team[bot] Jun 14, 2024
f63cb66
update components/pipeline-service/staging/base/kustomization.yaml
rh-tap-build-team[bot] Jun 14, 2024
3117971
update components/pipeline-service/staging/stone-stage-p01/deploy.yaml
rh-tap-build-team[bot] Jun 14, 2024
afa6a78
update components/pipeline-service/staging/stone-stg-m01/deploy.yaml
rh-tap-build-team[bot] Jun 14, 2024
b06ee22
update components/pipeline-service/staging/stone-stg-rh01/deploy.yaml
rh-tap-build-team[bot] Jun 14, 2024
134f999
KFLUXBUGS-863: introduce results fixes with requisite performance rel…
gabemontero Jun 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion components/monitoring/grafana/base/dashboards/pipeline-service/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://github.com/openshift-pipelines/pipeline-service/operator/gitops/argocd/grafana/?ref=72287aca6503f631b917debc27683a508f7e45ad
- https://github.com/openshift-pipelines/pipeline-service/operator/gitops/argocd/grafana/?ref=3ed2e97d4eacb0252b153edd2d6006637acb528e
2 changes: 1 addition & 1 deletion components/pipeline-service/development/bump-results-watcher-replicas.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
- op: replace
path: /spec/replicas
# default pipeline-service setting is 1
value: 2
value: 1
32 changes: 32 additions & 0 deletions components/pipeline-service/development/config.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
DB_USER=
DB_PASSWORD=
DB_HOST=
DB_PORT=5432
DB_NAME=
DB_SSLMODE=disable
DB_SSLROOTCERT=
DB_ENABLE_AUTO_MIGRATION=true
SERVER_PORT=8080
PROMETHEUS_PORT=9090
PROMETHEUS_HISTOGRAM=true
TLS_PATH=/etc/tls
AUTH_DISABLE=false
AUTH_IMPERSONATE=true
LOG_LEVEL=info
LOGS_API=false
LOGS_TYPE=File
LOGS_BUFFER_SIZE=5242880
LOGS_PATH=/logs
S3_BUCKET_NAME=
S3_ENDPOINT=
S3_HOSTNAME_IMMUTABLE=false
S3_REGION=
S3_ACCESS_KEY_ID=
S3_SECRET_ACCESS_KEY=
S3_MULTI_PART_SIZE=5242880
GCS_BUCKET_NAME=
STORAGE_EMULATOR_HOST=
K8S_QPS=50
K8S_BURST=100
PROFILING=true
PROFILING_PORT=6060
26 changes: 19 additions & 7 deletions components/pipeline-service/development/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,22 @@ commonAnnotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

resources:
- https://github.com/openshift-pipelines/pipeline-service.git/developer/openshift/gitops/argocd/pipeline-service?ref=72287aca6503f631b917debc27683a508f7e45ad
- https://github.com/openshift-pipelines/pipeline-service.git/developer/openshift/gitops/argocd/pipeline-service-storage?ref=72287aca6503f631b917debc27683a508f7e45ad
- https://github.com/openshift-pipelines/pipeline-service.git/developer/openshift/gitops/argocd/pipeline-service?ref=3ed2e97d4eacb0252b153edd2d6006637acb528e
- https://github.com/openshift-pipelines/pipeline-service.git/developer/openshift/gitops/argocd/pipeline-service-storage?ref=3ed2e97d4eacb0252b153edd2d6006637acb528e
- ../base/rbac

images:
- name: quay.io/redhat-appstudio/tekton-results-api
newTag: bae7851ff584423503af324200f52cd28ca99116
- name: quay.io/redhat-appstudio/tekton-results-watcher
newTag: bae7851ff584423503af324200f52cd28ca99116
# generate a new configmap with updated values (logs api, db ssl mode) and replace the default one
configMapGenerator:
- behavior: replace
files:
- config.env
name: api-config
options:
disableNameSuffixHash: true
- behavior: merge
name: config-observability
literals:
- profiling.enable="true"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we can get thread dumps if need be .... safe to run in prod per https://go.dev/doc/diagnostics

will not be turning on cpu profiling etc.

I see us turning on profiling for as many tekton controllers as possible in konflux as part of app sre reqs wrt thread dumps


patches:
- path: chains-tekton-config-patches.yaml
Expand Down Expand Up @@ -56,3 +63,8 @@ patches:
target:
kind: TektonConfig
name: config
- path: update-results-watcher-performance.yaml
target:
kind: Deployment
namespace: tekton-results
name: tekton-results-watcher
31 changes: 31 additions & 0 deletions components/pipeline-service/development/update-results-watcher-performance.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
- op: add
path: /spec/template/spec/containers/0/args/-
value: -threadiness
- op: add
path: /spec/template/spec/containers/0/args/-
value: "50"
- op: add
path: /spec/template/spec/containers/0/args/-
value: -qps
- op: add
path: /spec/template/spec/containers/0/args/-
value: "50"
- op: add
path: /spec/template/spec/containers/0/args/-
value: -burst
- op: add
path: /spec/template/spec/containers/0/args/-
value: "100"
- op: add
path: /spec/template/spec/containers/0/args/-
value: -update_log_timeout
- op: add
path: /spec/template/spec/containers/0/args/-
value: "9m"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on performance testing with @pmacik

- op: add
path: /spec/template/spec/containers/0/args/-
value: -dynamic_reconcile_timeout
- op: add
path: /spec/template/spec/containers/0/args/-
value: "9m"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on performance testing with @pmacik

2 changes: 1 addition & 1 deletion components/pipeline-service/staging/base/bump-results-watcher-replicas.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
- op: replace
path: /spec/replicas
# default pipeline-service setting is 1
value: 2
value: 1
32 changes: 32 additions & 0 deletions components/pipeline-service/staging/base/config.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
DB_USER=
DB_PASSWORD=
DB_HOST=
DB_PORT=5432
DB_NAME=
DB_SSLMODE=disable
DB_SSLROOTCERT=
DB_ENABLE_AUTO_MIGRATION=true
SERVER_PORT=8080
PROMETHEUS_PORT=9090
PROMETHEUS_HISTOGRAM=true
TLS_PATH=/etc/tls
AUTH_DISABLE=false
AUTH_IMPERSONATE=true
LOG_LEVEL=info
LOGS_API=false
LOGS_TYPE=File
LOGS_BUFFER_SIZE=5242880
LOGS_PATH=/logs
S3_BUCKET_NAME=
S3_ENDPOINT=
S3_HOSTNAME_IMMUTABLE=false
S3_REGION=
S3_ACCESS_KEY_ID=
S3_SECRET_ACCESS_KEY=
S3_MULTI_PART_SIZE=5242880
GCS_BUCKET_NAME=
STORAGE_EMULATOR_HOST=
K8S_QPS=50
K8S_BURST=100
PROFILING=true
PROFILING_PORT=6060
19 changes: 13 additions & 6 deletions components/pipeline-service/staging/base/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,24 @@ commonAnnotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

resources:
- https://github.com/openshift-pipelines/pipeline-service.git/operator/gitops/argocd/pipeline-service?ref=72287aca6503f631b917debc27683a508f7e45ad
- https://github.com/openshift-pipelines/pipeline-service.git/operator/gitops/argocd/pipeline-service?ref=3ed2e97d4eacb0252b153edd2d6006637acb528e
- pipelines-as-code-secret.yaml
- ../../base/external-secrets
- ../../base/testing
- ../../base/rbac

images:
- name: quay.io/redhat-appstudio/tekton-results-api
newTag: bae7851ff584423503af324200f52cd28ca99116
- name: quay.io/redhat-appstudio/tekton-results-watcher
newTag: bae7851ff584423503af324200f52cd28ca99116
# generate a new configmap with updated values (logs api, db ssl mode) and replace the default one
configMapGenerator:
- behavior: replace
files:
- config.env
name: api-config
options:
disableNameSuffixHash: true
- behavior: merge
name: config-observability
literals:
- profiling.enable="true"

patches:
- path: chains-tekton-config-patches.yaml
Expand Down
32 changes: 31 additions & 1 deletion components/pipeline-service/staging/base/update-results-watcher-performance.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,34 @@
value: "250m"
- op: replace
path: /spec/template/spec/containers/1/resources/limits/cpu
value: "250m"
value: "250m"
- op: add
path: /spec/template/spec/containers/1/args/-
value: -threadiness
- op: add
path: /spec/template/spec/containers/1/args/-
value: "50"
- op: add
path: /spec/template/spec/containers/1/args/-
value: -qps
- op: add
path: /spec/template/spec/containers/1/args/-
value: "50"
- op: add
path: /spec/template/spec/containers/1/args/-
value: -burst
- op: add
path: /spec/template/spec/containers/1/args/-
value: "100"
- op: add
path: /spec/template/spec/containers/1/args/-
value: -update_log_timeout
- op: add
path: /spec/template/spec/containers/1/args/-
value: "9m"
- op: add
path: /spec/template/spec/containers/1/args/-
value: -dynamic_reconcile_timeout
- op: add
path: /spec/template/spec/containers/1/args/-
value: "9m"
25 changes: 20 additions & 5 deletions components/pipeline-service/staging/stone-stage-p01/deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -897,7 +897,7 @@ subjects:
---
apiVersion: v1
data:
config.env: |
config.env: |-
DB_USER=
DB_PASSWORD=
DB_HOST=
Expand Down Expand Up @@ -926,6 +926,10 @@ data:
S3_MULTI_PART_SIZE=5242880
GCS_BUCKET_NAME=
STORAGE_EMULATOR_HOST=
K8S_QPS=50
K8S_BURST=100
PROFILING=true
PROFILING_PORT=6060
kind: ConfigMap
metadata:
annotations:
Expand Down Expand Up @@ -1054,6 +1058,7 @@ data:
metrics.taskrun.duration-type: "histogram"
metrics.pipelinerun.level: "pipeline"
metrics.pipelinerun.duration-type: "histogram"
profiling.enable: "true"
kind: ConfigMap
metadata:
annotations:
Expand Down Expand Up @@ -1335,7 +1340,7 @@ spec:
secretKeyRef:
key: db.name
name: tekton-results-database
image: quay.io/redhat-appstudio/tekton-results-api:bae7851ff584423503af324200f52cd28ca99116
image: quay.io/redhat-appstudio/tekton-results-api:ed360eccc021ad5eedf8ea9c0732912ef602b15a
livenessProbe:
httpGet:
path: /healthz
Expand Down Expand Up @@ -1404,7 +1409,7 @@ metadata:
name: tekton-results-watcher
namespace: tekton-results
spec:
replicas: 2
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: tekton-results-watcher
Expand Down Expand Up @@ -1470,6 +1475,16 @@ spec:
- -check_owner=false
- -completed_run_grace_period
- 10m
- -threadiness
- "50"
- -qps
- "50"
- -burst
- "100"
- -update_log_timeout
- 9m
- -dynamic_reconcile_timeout
- 9m
env:
- name: SYSTEM_NAMESPACE
valueFrom:
Expand All @@ -1487,7 +1502,7 @@ spec:
value: tekton-results-api-service.tekton-pipelines.svc.cluster.local:8080
- name: AUTH_MODE
value: token
image: quay.io/redhat-appstudio/tekton-results-watcher:bae7851ff584423503af324200f52cd28ca99116
image: quay.io/redhat-appstudio/tekton-results-watcher:ed360eccc021ad5eedf8ea9c0732912ef602b15a
name: watcher
ports:
- containerPort: 9090
Expand Down Expand Up @@ -1635,7 +1650,7 @@ spec:
)" \
--dry-run=client \
-o yaml | kubectl apply -f -
image: quay.io/konflux-ci/appstudio-utils:ab6b0b8e40e440158e7288c73aff1cf83a2cc8a9@sha256:24179f0efd06c65d16868c2d7eb82573cce8e43533de6cea14fec3b7446e0b14
image: quay.io/redhat-appstudio/appstudio-utils:dbbdd82734232e6289e8fbae5b4c858481a7c057
imagePullPolicy: Always
name: chains-secret-generation
resources:
Expand Down
25 changes: 20 additions & 5 deletions components/pipeline-service/staging/stone-stg-m01/deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -897,7 +897,7 @@ subjects:
---
apiVersion: v1
data:
config.env: |
config.env: |-
DB_USER=
DB_PASSWORD=
DB_HOST=
Expand Down Expand Up @@ -926,6 +926,10 @@ data:
S3_MULTI_PART_SIZE=5242880
GCS_BUCKET_NAME=
STORAGE_EMULATOR_HOST=
K8S_QPS=50
K8S_BURST=100
PROFILING=true
PROFILING_PORT=6060
kind: ConfigMap
metadata:
annotations:
Expand Down Expand Up @@ -1054,6 +1058,7 @@ data:
metrics.taskrun.duration-type: "histogram"
metrics.pipelinerun.level: "pipeline"
metrics.pipelinerun.duration-type: "histogram"
profiling.enable: "true"
kind: ConfigMap
metadata:
annotations:
Expand Down Expand Up @@ -1335,7 +1340,7 @@ spec:
secretKeyRef:
key: db.name
name: tekton-results-database
image: quay.io/redhat-appstudio/tekton-results-api:bae7851ff584423503af324200f52cd28ca99116
image: quay.io/redhat-appstudio/tekton-results-api:ed360eccc021ad5eedf8ea9c0732912ef602b15a
livenessProbe:
httpGet:
path: /healthz
Expand Down Expand Up @@ -1404,7 +1409,7 @@ metadata:
name: tekton-results-watcher
namespace: tekton-results
spec:
replicas: 2
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: tekton-results-watcher
Expand Down Expand Up @@ -1470,6 +1475,16 @@ spec:
- -check_owner=false
- -completed_run_grace_period
- 10m
- -threadiness
- "50"
- -qps
- "50"
- -burst
- "100"
- -update_log_timeout
- 9m
- -dynamic_reconcile_timeout
- 9m
env:
- name: SYSTEM_NAMESPACE
valueFrom:
Expand All @@ -1487,7 +1502,7 @@ spec:
value: tekton-results-api-service.tekton-pipelines.svc.cluster.local:8080
- name: AUTH_MODE
value: token
image: quay.io/redhat-appstudio/tekton-results-watcher:bae7851ff584423503af324200f52cd28ca99116
image: quay.io/redhat-appstudio/tekton-results-watcher:ed360eccc021ad5eedf8ea9c0732912ef602b15a
name: watcher
ports:
- containerPort: 9090
Expand Down Expand Up @@ -1635,7 +1650,7 @@ spec:
)" \
--dry-run=client \
-o yaml | kubectl apply -f -
image: quay.io/konflux-ci/appstudio-utils:ab6b0b8e40e440158e7288c73aff1cf83a2cc8a9@sha256:24179f0efd06c65d16868c2d7eb82573cce8e43533de6cea14fec3b7446e0b14
image: quay.io/redhat-appstudio/appstudio-utils:dbbdd82734232e6289e8fbae5b4c858481a7c057
imagePullPolicy: Always
name: chains-secret-generation
resources:
Expand Down
Loading
Loading