[bitnami/postgresql] Chart Version 12.2.1 leads to missing secret error

[jankoh]

Name and Version

bitnami/postgresql 12.2.1

What steps will reproduce the bug?

Upgrading to chart version 12.2.1 triggers a missing secret error:

ERROR:
  exit status 1
EXIT STATUS
  1
STDERR:
  Error: Failed to render chart: exit status 1: Error: execution error at (db-ai-tool/charts/postgresql-internal/templates/secrets.yaml:5:24): 
  PASSWORDS ERROR: You must provide your current passwords when upgrading the release.
                   Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.
                   Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases
      'global.postgresql.auth.postgresPassword' must not be empty, please add '--set global.postgresql.auth.postgresPassword=$POSTGRES_PASSWORD' to the command. To get the current value:
          export POSTGRES_PASSWORD=$(kubectl get secret --namespace "ai-tool-1322-testing" n4r-ai-tool-pgcredentials -o jsonpath="{.data.postgres-password}" | base64 -d)
  Use --debug flag to render out invalid YAML
  Error: plugin "diff" exited with error
COMBINED OUTPUT:
  Error: Failed to render chart: exit status 1: Error: execution error at (db-ai-tool/charts/postgresql-internal/templates/secrets.yaml:5:24): 
  PASSWORDS ERROR: You must provide your current passwords when upgrading the release.
                   Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.
                   Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases
      'global.postgresql.auth.postgresPassword' must not be empty, please add '--set global.postgresql.auth.postgresPassword=$POSTGRES_PASSWORD' to the command. To get the current value:
          export POSTGRES_PASSWORD=$(kubectl get secret --namespace "ai-tool-1322-testing" n4r-ai-tool-pgcredentials -o jsonpath="{.data.postgres-password}" | base64 -d)
  Use --debug flag to render out invalid YAML

The secrets are available in the namespace and contain the needed passwords; Chart-Version 12.1.15 is working using the identical configuration.

What I find extra remarkable: We got the chart update using renovate von Feb 17th; but the commit changing line 5 in secrets.yaml and setting Version to 12.2.1 is dated Feb 20th. Testing on a local Minikube leads to a different 12.2.1 chart having said line still using a hard-coded "postgres-password", which just works fine. Something weird seems to happen...

Are you using any custom parameters or values?

Config used to deploy:

postgresql-internal:
  global:
    postgresql:
      auth:
        existingSecret: n4r-ai-tool-pgcredentials
        database: "ai-tool"
        username: "technicalUser"
  image:
    tag: "14"
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true

What is the expected behavior?

Deployment should succeed.

What do you see instead?

Deployment fails.

Additional information

No response

[javsalgar]

Hi!

From which version of the chart are you trying to upgrade? It seems to be an issue coming from the old version.

[jankoh]

Hi, we're upgrading from 12.1.15. I downgraded our dev deployment back to that version after the failing upgrade.

[rmoreas]

We got the same issue doing update from helm chart version 12.1.13 to 12.2.0

[jotamartos]

Hi all,

I just tried to reproduce the issue but I couldn't do so. I installed version 12.1.15 of the chart and waited for it to be ready. I accessed the database and confirmed I could access PostgreSQL with the provided password. After that, I upgraded the solution to the latest version and didn't get any error

helm install jota-postgresql bitnami/postgresql --version 12.1.15
kubectl get pods -w
helm upgrade jota-postgresql bitnami/postgresql

I confirmed I didn't have any previous PVs or PVCs when installing the chart for the first time. Can you check if this configuration (default parameters) work for you?

postgresql-internal:
  global:
    postgresql:
      auth:
        existingSecret: n4r-ai-tool-pgcredentials
        database: "ai-tool"
        username: "technicalUser"

@jankoh, just to confirm, could you please let us know how you created that secret and its content (you can add example password values there) so we confirm everything is as expected?

[jankoh]

@jotamartos the secret looks the following (passwords replaced):

apiVersion: v1
data:
  password: password1
  postgres-password: password2
  replication-password: password3
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: ai-tool
    meta.helm.sh/release-namespace: ai-tool-1322-testing
  creationTimestamp: "2023-01-02T15:06:24Z"
  labels:
    app.kubernetes.io/instance: ai-tool
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: db-ai-tool
    helm.sh/chart: db-ai-tool-0.2.0
  name: n4r-ai-tool-pgcredentials
  namespace: ai-tool-1322-testing
  resourceVersion: "139353018"
  uid: 3da4ffca-5033-443a-8520-d564679fcd7e
type: Opaque

As written: If I try to reproduce the issue locally, I get a "different version" 12.2.1 of the chart; not having the contents of mentioned PR. Using this altered 12.2.1 I can deploy locally; however our deployment routines download the repos when deploying, and I get the error in that case.

The file secrets.yaml, line 5, locally looks like that:

{{- $postgresPassword = include "common.secrets.passwords.manage" (dict "secret" (include "postgresql.secretName" .) "key" "postgres-password" "providedValues" (list "global.postgresql.auth.postgresPassword" "auth.postgresPassword") "context" $) | trimAll "\"" | b64dec }}

which clearly is before merging #15023. However, the PR changes version 12.2.0 to 12.2.1, so there seems to be something weird gong on.

[jankoh]

Hi,

we just tested Version 12.2.2 just released. The issue is still present, but we figured the only thing failing is the helm diff step. If we omit a diff, the deployment succeeds.

[jotamartos]

Hi,

Sorry for the delay here . I tried to reproduce the issue but I couldn't do so. The steps I followed were the following ones:

• Create a secret

k create secret generic n4r-ai-tool-pgcredentials --from-literal=password=password1 --from-literal=postgres-password=password2 --from-literal=replication-password=password3

• Edit the default configuration

$ git diff values.yaml
diff --git a/bitnami/postgresql/values.yaml b/bitnami/postgresql/values.yaml
index f770b9878..a137f2d3d 100644
--- a/bitnami/postgresql/values.yaml
+++ b/bitnami/postgresql/values.yaml
@@ -26,10 +26,10 @@ global:
     ##
     auth:
       postgresPassword: ""
-      username: ""
+      username: "technicalUser"
       password: ""
-      database: ""
-      existingSecret: ""
+      database: "ai-tool"
+      existingSecret: "n4r-ai-tool-pgcredentials"
       secretKeys:
         adminPasswordKey: ""
         userPasswordKey: ""

• Launch PostgreSQL 12.1.15 with the previous changes

$ helm install jota-postgresql -f values.yaml --version 12.1.15 bitnami/postgresql
NAME: jota-postgresql
LAST DEPLOYED: Tue Mar  7 08:28:34 2023
NAMESPACE: jotamartos
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: postgresql
CHART VERSION: 12.1.15
APP VERSION: 15.2.0

** Please be patient while the chart is being deployed **

PostgreSQL can be accessed via port 5432 on the following DNS names from within your cluster:

    jota-postgresql.jotamartos.svc.cluster.local - Read/Write connection

To get the password for "postgres" run:

    export POSTGRES_ADMIN_PASSWORD=$(kubectl get secret --namespace jotamartos n4r-ai-tool-pgcredentials -o jsonpath="{.data.postgres-password}" | base64 -d)

To get the password for "technicalUser" run:

    export POSTGRES_PASSWORD=$(kubectl get secret --namespace jotamartos n4r-ai-tool-pgcredentials -o jsonpath="{.data.password}" | base64 -d)

• Confirm I could access the database using the postgres and technicalUser users and the passwords I set in the secret.
• Upgrade the deployment

$ helm upgrade jota-postgresql bitnami/postgresql -f values.yaml
Release "jota-postgresql" has been upgraded. Happy Helming!
NAME: jota-postgresql
LAST DEPLOYED: Tue Mar  7 08:32:01 2023
NAMESPACE: jotamartos
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
CHART NAME: postgresql
CHART VERSION: 12.2.2
APP VERSION: 15.2.0

** Please be patient while the chart is being deployed **

PostgreSQL can be accessed via port 5432 on the following DNS names from within your cluster:

    jota-postgresql.jotamartos.svc.cluster.local - Read/Write connection

To get the password for "postgres" run:

    export POSTGRES_ADMIN_PASSWORD=$(kubectl get secret --namespace jotamartos n4r-ai-tool-pgcredentials -o jsonpath="{.data.postgres-password}" | base64 -d)

To get the password for "technicalUser" run:

    export POSTGRES_PASSWORD=$(kubectl get secret --namespace jotamartos n4r-ai-tool-pgcredentials -o jsonpath="{.data.password}" | base64 -d)

• Confirm I could access the database again

I have no name!@jota-postgresql-0:/$ psql -U technicalUser ai-tool
Password for user technicalUser:
psql (15.2)
Type "help" for help.

ai-tool=> \l
                                                         List of databases
   Name    |     Owner     | Encoding |   Collate   |    Ctype    | ICU Locale | Locale Provider |        Access privileges
-----------+---------------+----------+-------------+-------------+------------+-----------------+---------------------------------
 ai-tool   | technicalUser | UTF8     | en_US.UTF-8 | en_US.UTF-8 |            | libc            | =Tc/technicalUser              +
           |               |          |             |             |            |                 | technicalUser=CTc/technicalUser
 postgres  | postgres      | UTF8     | en_US.UTF-8 | en_US.UTF-8 |            | libc            |
 template0 | postgres      | UTF8     | en_US.UTF-8 | en_US.UTF-8 |            | libc            | =c/postgres                    +
           |               |          |             |             |            |                 | postgres=CTc/postgres
 template1 | postgres      | UTF8     | en_US.UTF-8 | en_US.UTF-8 |            | libc            | =c/postgres                    +
           |               |          |             |             |            |                 | postgres=CTc/postgres
(4 rows)

ai-tool=> \q

As you can see, I didn't get any error in any of the steps. Can you confirm you are following these steps?

[jankoh]

Hi again,

as written in the previous comment: the main culprit is the diff step - at least in our deployment. If we simply do a helm upgrade without a diff, the whole thing works. So it's probably a bug or missing feature in the helm diff plugin.

We'll move on without the diff for now and are going to use helm secrets in the near future so the "existingSecret" setting will become obsolete for our deployments.

[jotamartos]

Sorry, didn't notice that comment. I'm glad to hear that the deployment works as expected. The issue is probably a bug in the diff plugin, you can let them know about this so they take a look at it.

Thanks!

[LGLN-LS]

I have the same issue with chart version 12.2.x, helm upgrade does not work anymore with an existingSecret.

But I noticed, that if I do helm upgrade mydeployment bitnami/postgresql -f values.yaml --set global.postgresql.auth.postgresPassword=NotMyPassword --dry-run it does work.

I mean if I set global.postgresql.auth.postgresPassword to any string, it will bypass the missing password check, but the helm chart will correctly use my password from existingSecret everywhere and deploy successfully. That password is not put into the outgoing template.

In my opinion, there is something wrong with the password check, it does not respect if existingSecret is set.

[rmoreas]

@rmoreas Same for us (helmfile). Try HELMFILE_MODE=sync. That fixes the issue for us, since it omits the diff.

Didn't know about this "HELMFILE_MODE". So then you basically run "helmfile sync"?

As I understand it well, this will create a new helm release on each run, even if there are no changes. That's not what we want.

[jankoh]

Hey @rmoreas there is (AFAIR) a bug report for the helm diff plugin. The main point here is the mentioned change making the configuration variable names variables themselfs. IIRC, that is what breaks helm diff for some reason. I do agree, the sync is probably not the best solution, it's more a workaround. We aim using helm secrets, specify the secrets directly, and stepping back from using the "existingSecret" configuration in the near future to be able to use diff, again. This is bit of a larger change, so it's not yet finished...

[LGLN-LS]

@jotamartos I do not use any helm plugin. I just do a helm upgrade.
My auth settings:

auth:
  existingSecret: aks-testdb-postgresql
  database: aks-testdb

If I do a helm install it works, but I get this message:

WARNING: The configured password will be ignored on new installation in case when previous Posgresql release was deleted through the helm command. In that case, old PVC will have an old password, and setting it through helm won't take effect. Deleting persistent volumes (PVs) will solve the issue.

and after an helm upgrade this error happens:

Error: UPGRADE FAILED: execution error at (postgresql/templates/secrets.yaml:5:24):
PASSWORDS ERROR: You must provide your current passwords when upgrading the release.
                 Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.
                 Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases

    'global.postgresql.auth.postgresPassword' must not be empty, please add '--set global.postgresql.auth.postgresPassword=$POSTGRES_PASSWORD' to the command. To get the current value:

        export POSTGRES_PASSWORD=$(kubectl get secret --namespace "aks-testdb" aks-testdb-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)

I understand that if you let the chart auto generate the password, that you have to provide it. But I specified a fixed password in a secret and as I already said it works fine if I bypass the password check.

[jotamartos]

Hi @LGLN-LS,

From which version to which one are you upgrading? In my test, I upgraded from 12.1.15 to 12.2.2 (the latest one at that moment)? As you can see there, I didn't get any error and I also created a secret with the password.

[LGLN-LS]

@jotamartos for the purpose of complete testing I created a new database:

> kubectl create namespace dbtest
> kubectl create secret generic mycredentials --from-literal=password=password1 --from-literal=postgres-password=password2 -n dbtest

> helm install dbtest bitnami/postgresql --version "~12.1.0" -n dbtest --set auth.existingSecret=mycredentials --set auth.database=dbtest --set volumePermissions.enabled=true --set primary.persistence.storageClass=ibmc-block-gold
NAME: dbtest
LAST DEPLOYED: Tue Apr  4 10:32:26 2023
NAMESPACE: dbtest
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: postgresql
CHART VERSION: 12.1.15
APP VERSION: 15.2.0

** Please be patient while the chart is being deployed **

PostgreSQL can be accessed via port 5432 on the following DNS names from within your cluster:

    dbtest-postgresql.dbtest.svc.cluster.local - Read/Write connection

To get the password for "postgres" run:

    export POSTGRES_PASSWORD=$(kubectl get secret --namespace dbtest mycredentials -o jsonpath="{.data.postgres-password}" | base64 -d)

To connect to your database run the following command:

    kubectl run dbtest-postgresql-client --rm --tty -i --restart='Never' --namespace dbtest --image docker.io/bitnami/postgresql:15.2.0-debian-11-r0 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
      --command -- psql --host dbtest-postgresql -U postgres -d dbtest -p 5432

    > NOTE: If you access the container using bash, make sure that you execute "/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash" in order to avoid the error "psql: local user with ID 1001} does not exist"

To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace dbtest svc/dbtest-postgresql 5432:5432 &
    PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d dbtest -p 5432

WARNING: The configured password will be ignored on new installation in case when previous Posgresql release was deleted through the helm command. In that case, old PVC will have an old password, and setting it through helm won't take effect. Deleting persistent volumes (PVs) will solve the issue.

Then I tried to upgrade:

> helm upgrade dbtest bitnami/postgresql --version "~12.2.0" -n dbtest --set auth.existingSecret=mycredentials --set auth.database=dbtest --set volumePermissions.enabled=true --set primary.persistence.storageClass=ibmc-block-gold
Error: UPGRADE FAILED: execution error at (postgresql/templates/secrets.yaml:5:24):
PASSWORDS ERROR: You must provide your current passwords when upgrading the release.
                 Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.
                 Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases

    'global.postgresql.auth.postgresPassword' must not be empty, please add '--set global.postgresql.auth.postgresPassword=$POSTGRES_PASSWORD' to the command. To get the current value:

        export POSTGRES_PASSWORD=$(kubectl get secret --namespace "dbtest" mycredentials -o jsonpath="{.data.postgres-password}" | base64 -d)

[jotamartos]

Sorry, but there should be something wrong in your environment. I tried to follow your steps and everything worked as expected

$ helm upgrade dbtest bitnami/postgresql --version "~12.2.0" --set auth.existingSecret=mycredentials --set auth.database=dbtest
Release "dbtest" has been upgraded. Happy Helming!
NAME: dbtest
LAST DEPLOYED: Tue Apr 11 14:17:27 2023
NAMESPACE: jotamartos
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
CHART NAME: postgresql
CHART VERSION: 12.2.7
APP VERSION: 15.2.0

** Please be patient while the chart is being deployed **

PostgreSQL can be accessed via port 5432 on the following DNS names from within your cluster:

    dbtest-postgresql.jotamartos.svc.cluster.local - Read/Write connection

To get the password for "postgres" run:

    export POSTGRES_PASSWORD=$(kubectl get secret --namespace jotamartos mycredentials -o jsonpath="{.data.postgres-password}" | base64 -d)

To connect to your database run the following command:

    kubectl run dbtest-postgresql-client --rm --tty -i --restart='Never' --namespace jotamartos --image docker.io/bitnami/postgresql:15.2.0-debian-11-r16 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
      --command -- psql --host dbtest-postgresql -U postgres -d dbtest -p 5432

    > NOTE: If you access the container using bash, make sure that you execute "/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash" in order to avoid the error "psql: local user with ID 1001} does not exist"

To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace jotamartos svc/dbtest-postgresql 5432:5432 &
    PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d dbtest -p 5432

WARNING: The configured password will be ignored on new installation in case when previous Posgresql release was deleted through the helm command. In that case, old PVC will have an old password, and setting it through helm won't take effect. Deleting persistent volumes (PVs) will solve the issue.

I also confirmed that the database is created and that I can access the database with the provided credentials.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[morremeyer]

For anyone interested in a permanent fix: I opened databus23/helm-diff#460 so that we can maybe get to the source of the issue with helm diff here.