feat: added an optional cronjob to perform the dbclean (external trig…

…ger) (#52) Signed-off-by: Sherif Ayad <[email protected]>
pact-foundation · Apr 7, 2023 · 37a7833 · 37a7833
1 parent 5e75ac2
commit 37a7833
Show file tree

Hide file tree

Showing 7 changed files with 147 additions and 75 deletions.
diff --git a/charts/pact-broker/Chart.yaml b/charts/pact-broker/Chart.yaml
@@ -2,7 +2,7 @@ apiVersion: v2
 name: pact-broker
 description: The Pact Broker is an application for sharing for Pact contracts and verification results.
 type: application
-version: 0.9.0
+version: 0.10.0
 appVersion: 2.105.0.1
 dependencies:
   - condition: postgresql.enabled

diff --git a/charts/pact-broker/README.md b/charts/pact-broker/README.md
@@ -1,6 +1,6 @@
 # pact-broker
 
-![Version: 0.9.0](https://img.shields.io/badge/Version-0.9.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.105.0.1](https://img.shields.io/badge/AppVersion-2.105.0.1-informational?style=flat-square)
+![Version: 0.10.0](https://img.shields.io/badge/Version-0.10.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.105.0.1](https://img.shields.io/badge/AppVersion-2.105.0.1-informational?style=flat-square)
 
 The Pact Broker is an application for sharing for Pact contracts and verification results.
 
@@ -86,8 +86,9 @@ helm upgrade -i <release_name> oci://ghcr.io/pact-foundation/pact-broker-chart/p
 | broker.config.databaseClean.cronSchedule | Set to a cron schedule that will run when your Broker is under the least operational load. | string | `"15 2 * * *"` |
 | broker.config.databaseClean.deletionLimit | The maximum number of records to delete at a time for each of the removable data categories. | int | `500` |
 | broker.config.databaseClean.dryRun | Defaults to false. Set to true to see the output of what would have been deleted if the task had run. | bool | `false` |
-| broker.config.databaseClean.enabled | Set to true to enable the automatic data clean up. | bool | `false` |
-| broker.config.databaseClean.keepVersionSelectors | A JSON string containing a list of the "keep" selectors. | string | `"[{ \"max_age\": 180 }]"` |
+| broker.config.databaseClean.enabled | Set to true to enable the automatic data cleanup. | bool | `false` |
+| broker.config.databaseClean.keepVersionSelectors | A JSON string containing a list of the "keep" selectors. | string | `"[{\"latest\": true}, { \"max_age\": 180 }]"` |
+| broker.config.databaseClean.mode | Set the mode of the cleanup task. Can either be `embedded` or `external`. Setting the mode to `external` will create a Kubernetes `CronJob` to handle the cleanup; thus implementing https://docs.pact.io/pact_broker/docker_images/pactfoundation#running-the-clean-task-from-an-external-source | string | `"embedded"` |
 | broker.config.databaseClean.overwrittenDataMaxAge | The maximum number of days to keep "overwritten" data. | int | `90` |
 | broker.config.databaseConnectMaxRetries | Setting the max retries to a non-zero number will allow it to retry the connection the configured number of times, waiting 3 seconds between attempts. | int | `0` |
 | broker.config.databaseConnectionValidationTimeout |  | int | `3600` |
@@ -225,3 +226,10 @@ Configure the Pact Broker by using the username credential that you configure vi
 
 #### Specify Credentials via Secret
 Configure the Pact Broker to use an existing Secret to retrieve the user password as a means to connect to the database. Ensure that the Kubernetes Secret has the password in the `user-password` field and ensure that you have set `externalDatabase.config.auth.existingSecret` value to the name of the secret. To configure the username, you can use the `username` value.
+
+### Database Clean Task
+Pact Broker [automatic data cleanup](https://docs.pact.io/pact_broker/docker_images/pactfoundation#automatic-data-clean-up) can be enabled by setting the property `broker.config.databaseClean.enabled` to `true`.
+
+By default and for simple installations, one could rely on the built-in integrated mode of the cleanup (the property `broker.config.databaseClean.mode` having the value `embedded`).
+
+For a more advanced setup e.g. in highly available installations with more than one replica, one should rely on the [external source pattern](https://docs.pact.io/pact_broker/docker_images/pactfoundation#running-the-clean-task-from-an-external-source) of running the clean task. That can be achieved by setting the property `broker.config.databaseClean.mode` to external, which would result in having a `CronJob` performing the cleanup task instead of the main application Pods.
diff --git a/charts/pact-broker/README.md.gotmpl b/charts/pact-broker/README.md.gotmpl
@@ -88,3 +88,10 @@ Configure the Pact Broker by using the username credential that you configure vi
 
 #### Specify Credentials via Secret
 Configure the Pact Broker to use an existing Secret to retrieve the user password as a means to connect to the database. Ensure that the Kubernetes Secret has the password in the `user-password` field and ensure that you have set `externalDatabase.config.auth.existingSecret` value to the name of the secret. To configure the username, you can use the `username` value.
+
+### Database Clean Task
+Pact Broker [automatic data cleanup](https://docs.pact.io/pact_broker/docker_images/pactfoundation#automatic-data-clean-up) can be enabled by setting the property `broker.config.databaseClean.enabled` to `true`.
+
+By default and for simple installations, one could rely on the built-in integrated mode of the cleanup (the property `broker.config.databaseClean.mode` having the value `embedded`).
+
+For a more advanced setup e.g. in highly available installations with more than one replica, one should rely on the [external source pattern](https://docs.pact.io/pact_broker/docker_images/pactfoundation#running-the-clean-task-from-an-external-source) of running the clean task. That can be achieved by setting the property `broker.config.databaseClean.mode` to external, which would result in having a `CronJob` performing the cleanup task instead of the main application Pods.
diff --git a/charts/pact-broker/templates/_helpers.tpl b/charts/pact-broker/templates/_helpers.tpl
@@ -122,11 +122,92 @@ Return the databaseSecret key to retrieve credentials for database
 
 {{/*
  Create the name of the service account to use
- */}}
+*/}}
 {{- define "broker.serviceAccountName" -}}
 {{- if .Values.serviceAccount.create -}}
     {{ default (include "common.names.fullname" .) .Values.serviceAccount.name }}
 {{- else -}}
     {{ default "default" .Values.serviceAccount.name }}
 {{- end -}}
+{{- end -}}
+
+{{/*
+Logging ENV Vars
+*/}}
+{{ define "envVars.logging" }}
+- name: PACT_BROKER_LOG_LEVEL
+  value: {{ .Values.broker.config.logLevel | quote }}
+- name: PACT_BROKER_LOG_FORMAT
+  value: {{ .Values.broker.config.logFormat | quote }}
+- name: PACT_BROKER_HTTP_DEBUG_LOGGING_ENABLED
+  value: {{ .Values.broker.config.httpDebugLoggingEnabled | quote }}
+- name: PACT_BROKER_HIDE_PACTFLOW_MESSAGES
+  value: {{ .Values.broker.config.hidePactflowMessages | quote }}
+{{- end -}}
+
+{{/*
+Database ENV Vars
+*/}}
+{{- define "envVars.db" -}}
+- name: PACT_BROKER_DATABASE_ADAPTER
+  value: {{ include "broker.databaseAdapter" . }}
+- name: PACT_BROKER_DATABASE_HOST
+  value: {{ include "broker.databaseHost" . }}
+- name: PACT_BROKER_DATABASE_PORT
+  value: {{ include "broker.databasePort" . }}
+- name: PACT_BROKER_DATABASE_NAME
+  value: {{ include "broker.databaseName" . }}
+- name: PACT_BROKER_DATABASE_USERNAME
+  value: {{ include "broker.databaseUser" . }}
+- name: PACT_BROKER_DATABASE_PASSWORD
+  {{- if and .Values.postgresql.enabled .Values.postgresql.auth.password }}
+  value: {{ .Values.postgresql.auth.password | quote }}
+  {{- else if and .Values.externalDatabase.enabled .Values.externalDatabase.config.auth.password }}
+  value: {{ .Values.externalDatabase.config.auth.password | quote }}
+  {{- else }}
+  valueFrom:
+    secretKeyRef:
+      name: {{ include "broker.databaseSecretName" . }}
+      key: {{ include "broker.databaseSecretKey" . }}
+  {{- end }}
+- name: PACT_BROKER_DATABASE_SSLMODE
+  value: {{ .Values.broker.config.databaseSslmode | quote }}
+- name: PACT_BROKER_SQL_LOG_LEVEL
+  value: {{ .Values.broker.config.sqlLogLevel | quote }}
+- name: PACT_BROKER_SQL_LOG_WARN_DURATION
+  value: {{ .Values.broker.config.sqlLogWarnDuration | quote }}
+- name: PACT_BROKER_SQL_ENABLE_CALLER_LOGGING
+  value: {{ .Values.broker.config.sqlEnableCallerLogging | quote }}
+- name: PACT_BROKER_DATABASE_MAX_CONNECTIONS
+  value: {{ .Values.broker.config.databaseMaxConnections | quote }}
+- name: PACT_BROKER_DATABASE_POOL_TIMEOUT
+  value: {{ .Values.broker.config.databasePoolTimeout | quote }}
+- name: PACT_BROKER_DATABASE_CONNECT_MAX_RETRIES
+  value: {{ .Values.broker.config.databaseConnectMaxRetries | quote }}
+- name: PACT_BROKER_AUTO_MIGRATE_DB
+  value: {{ .Values.broker.config.autoMigrateDb | quote }}
+- name: PACT_BROKER_AUTO_MIGRATE_DB_DATA
+  value: {{ .Values.broker.config.autoMigrateDbData | quote }}
+- name: PACT_BROKER_ALLOW_MISSING_MIGRATION_FILES
+  value: {{ .Values.broker.config.allowMissingMigrationFiles | quote }}
+- name: PACT_BROKER_DATABASE_STATEMENT_TIMEOUT
+  value: {{ .Values.broker.config.databaseStatementTimeout | quote }}
+- name: PACT_BROKER_METRICS_SQL_STATEMENT_TIMEOUT
+  value: {{ .Values.broker.config.metricsSqlStatementTimeout | quote }}
+- name: PACT_BROKER_DATABASE_CONNECTION_VALIDATION_TIMEOUT
+  value: {{ .Values.broker.config.databaseConnectionValidationTimeout | quote }}
+{{- end -}}
+
+{{/*
+Database Cleanup ENV Vars
+*/}}
+{{- define "envVars.dbCleanup" -}}
+- name: PACT_BROKER_DATABASE_CLEAN_DELETION_LIMIT
+  value: {{ .Values.broker.config.databaseClean.deletionLimit | quote }}
+- name: PACT_BROKER_DATABASE_CLEAN_OVERWRITTEN_DATA_MAX_AGE
+  value: {{ .Values.broker.config.databaseClean.overwrittenDataMaxAge | quote }}
+- name: PACT_BROKER_DATABASE_CLEAN_KEEP_VERSION_SELECTORS
+  value: {{ .Values.broker.config.databaseClean.keepVersionSelectors | quote }}
+- name: PACT_BROKER_DATABASE_CLEAN_DRY_RUN
+  value: {{ .Values.broker.config.databaseClean.dryRun | quote }}
 {{- end -}}
diff --git a/charts/pact-broker/templates/cronjob.yml b/charts/pact-broker/templates/cronjob.yml
@@ -0,0 +1,33 @@
+{{- if and (.Values.broker.config.databaseClean.enabled) (eq .Values.broker.config.databaseClean.mode "external") }}
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: {{ include "common.names.fullname" . }}-dbclean
+  namespace: {{ .Release.Namespace | quote }}
+  labels:
+    name: {{ include "chart.fullname" . }}
+    app.kubernetes.io/name: {{ include "chart.name" . }}
+    helm.sh/chart: {{ .Chart.Name }}
+    app.kubernetes.io/instance: {{ .Release.Name }}
+spec:
+  schedule: {{ .Values.broker.config.databaseClean.cronSchedule | quote }}
+  concurrencyPolicy: Forbid
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          containers:
+            - name: {{ template "chart.fullname" . }}-dbclean
+              image: {{ template "broker.image" . }}
+              imagePullPolicy: {{ .Values.image.pullPolicy }}
+              command:
+                - clean
+              env:
+                # Logging
+                {{- include "envVars.logging" . | nindent 16 }}
+                # Database
+                {{- include "envVars.db" . | nindent 16 }}
+                # Database clean up (external mode)
+                {{- include "envVars.dbCleanup" . | nindent 16 }}
+          restartPolicy: Never
+{{- end}}
diff --git a/charts/pact-broker/templates/deployment.yaml b/charts/pact-broker/templates/deployment.yaml
@@ -57,78 +57,18 @@ spec:
               containerPort: {{ .Values.broker.containerPorts.https }}
               protocol: TCP
           env:
-            # Database clean up
-            {{- if .Values.broker.config.databaseClean.enabled }}
+            # Logging
+            {{- include "envVars.logging" . | nindent 12 }}
+            # Database
+            {{- include "envVars.db" . | nindent 12 }}
+            # Database Cleanup (embedded mode)
+            {{- if and (.Values.broker.config.databaseClean.enabled) (eq .Values.broker.config.databaseClean.mode "embedded") }}
             - name: PACT_BROKER_DATABASE_CLEAN_ENABLED
               value: {{ .Values.broker.config.databaseClean.enabled | quote }}
             - name: PACT_BROKER_DATABASE_CLEAN_CRON_SCHEDULE
               value: {{ .Values.broker.config.databaseClean.cronSchedule | quote }}
-            - name: PACT_BROKER_DATABASE_CLEAN_DELETION_LIMIT
-              value: {{ .Values.broker.config.databaseClean.deletionLimit | quote }}
-            - name: PACT_BROKER_DATABASE_CLEAN_OVERWRITTEN_DATA_MAX_AGE
-              value: {{ .Values.broker.config.databaseClean.overwrittenDataMaxAge | quote }}
-            - name: PACT_BROKER_DATABASE_CLEAN_KEEP_VERSION_SELECTORS
-              value: {{ .Values.broker.config.databaseClean.keepVersionSelectors | quote }}
-            - name: PACT_BROKER_DATABASE_CLEAN_DRY_RUN
-              value: {{ .Values.broker.config.databaseClean.dryRun | quote }}
-            {{- end}}
-            # Logging
-            - name: PACT_BROKER_LOG_LEVEL
-              value: {{ .Values.broker.config.logLevel | quote }}
-            - name: PACT_BROKER_LOG_FORMAT
-              value: {{ .Values.broker.config.logFormat | quote }}
-            - name: PACT_BROKER_HTTP_DEBUG_LOGGING_ENABLED
-              value: {{ .Values.broker.config.httpDebugLoggingEnabled | quote }}
-            - name: PACT_BROKER_HIDE_PACTFLOW_MESSAGES
-              value: {{ .Values.broker.config.hidePactflowMessages | quote }}
-            # Database
-            - name: PACT_BROKER_DATABASE_ADAPTER
-              value: {{ include "broker.databaseAdapter" . }}
-            - name: PACT_BROKER_DATABASE_HOST
-              value: {{ include "broker.databaseHost" . }}
-            - name: PACT_BROKER_DATABASE_PORT
-              value: {{ include "broker.databasePort" . }}
-            - name: PACT_BROKER_DATABASE_NAME
-              value: {{ include "broker.databaseName" . }}
-            - name: PACT_BROKER_DATABASE_USERNAME
-              value: {{ include "broker.databaseUser" . }}
-            - name: PACT_BROKER_DATABASE_PASSWORD
-              {{- if and .Values.postgresql.enabled .Values.postgresql.auth.password }}
-              value: {{ .Values.postgresql.auth.password | quote }}
-              {{- else if and .Values.externalDatabase.enabled .Values.externalDatabase.config.auth.password }}
-              value: {{ .Values.externalDatabase.config.auth.password | quote }}
-              {{- else }}
-              valueFrom:
-                secretKeyRef:
-                  name: {{ include "broker.databaseSecretName" . }}
-                  key: {{ include "broker.databaseSecretKey" . }}
-              {{- end }}
-            - name: PACT_BROKER_DATABASE_SSLMODE
-              value: {{ .Values.broker.config.databaseSslmode | quote }}
-            - name: PACT_BROKER_SQL_LOG_LEVEL
-              value: {{ .Values.broker.config.sqlLogLevel | quote }}
-            - name: PACT_BROKER_SQL_LOG_WARN_DURATION
-              value: {{ .Values.broker.config.sqlLogWarnDuration | quote }}
-            - name: PACT_BROKER_SQL_ENABLE_CALLER_LOGGING
-              value: {{ .Values.broker.config.sqlEnableCallerLogging | quote }}
-            - name: PACT_BROKER_DATABASE_MAX_CONNECTIONS
-              value: {{ .Values.broker.config.databaseMaxConnections | quote }}
-            - name: PACT_BROKER_DATABASE_POOL_TIMEOUT
-              value: {{ .Values.broker.config.databasePoolTimeout | quote }}
-            - name: PACT_BROKER_DATABASE_CONNECT_MAX_RETRIES
-              value: {{ .Values.broker.config.databaseConnectMaxRetries | quote }}
-            - name: PACT_BROKER_AUTO_MIGRATE_DB
-              value: {{ .Values.broker.config.autoMigrateDb | quote }}
-            - name: PACT_BROKER_AUTO_MIGRATE_DB_DATA
-              value: {{ .Values.broker.config.autoMigrateDbData | quote }}
-            - name: PACT_BROKER_ALLOW_MISSING_MIGRATION_FILES
-              value: {{ .Values.broker.config.allowMissingMigrationFiles | quote }}
-            - name: PACT_BROKER_DATABASE_STATEMENT_TIMEOUT
-              value: {{ .Values.broker.config.databaseStatementTimeout | quote }}
-            - name: PACT_BROKER_METRICS_SQL_STATEMENT_TIMEOUT
-              value: {{ .Values.broker.config.metricsSqlStatementTimeout | quote }}
-            - name: PACT_BROKER_DATABASE_CONNECTION_VALIDATION_TIMEOUT
-              value: {{ .Values.broker.config.databaseConnectionValidationTimeout | quote }}
+            {{- include "envVars.dbCleanup" . | nindent 12 }}
+            {{- end }}
             # Authentication and authorization
             - name: PACT_BROKER_BASIC_AUTH_ENABLED
               value: {{ .Values.broker.config.basicAuth.enabled | quote }}

diff --git a/charts/pact-broker/values.yaml b/charts/pact-broker/values.yaml
@@ -258,9 +258,12 @@ broker:
     # Pact Broker [automatic data cleanup](https://docs.pact.io/pact_broker/docker_images/pactfoundation#automatic-data-clean-up)
     databaseClean:
 
-      # -- Set to true to enable the automatic data clean up.
+      # -- Set to true to enable the automatic data cleanup.
       enabled: false
 
+      # -- Set the mode of the cleanup task. Can either be `embedded` or `external`. Setting the mode to `external` will create a Kubernetes `CronJob` to handle the cleanup; thus implementing https://docs.pact.io/pact_broker/docker_images/pactfoundation#running-the-clean-task-from-an-external-source
+      mode: embedded
+
       # -- Set to a cron schedule that will run when your Broker is under the least operational load.
       cronSchedule: 15 2 * * *
 
@@ -271,7 +274,7 @@ broker:
       overwrittenDataMaxAge: 90
 
       # -- A JSON string containing a list of the "keep" selectors.
-      keepVersionSelectors: '[{ "max_age": 180 }]'
+      keepVersionSelectors: '[{"latest": true}, { "max_age": 180 }]'
 
       # -- Defaults to false. Set to true to see the output of what would have been deleted if the task had run.
       dryRun: false