GCS service account credential doesn't format correctly when using ENV var

[zakris-s]

Describe the bug

When using an environment variable for the GCS service account key JSON data Mimir pods that depend on that credential won't start with the error code err="invalid character 't' looking for beginning of value failed to create credentials from JSON.

To Reproduce

Steps to reproduce the behavior:

1. Specify an ENV var contained GCS service account JSON data like this:

common:
  storage:
     backend: gcs
      gcs:
        service_account: |
          ${GCS_SA_VAR_NAME}

2. Install Mimir via the helm chart

Expected behavior

Mimir will start normally and use the above VAR as the GCS credentials to access the bucket

Environment

• Infrastructure: Kubernetes GKE
• Deployment tool: Helm

Additional Context

Note: If I copy the ENV var from one of the pods and paste it in to the above config it works as expected.

[narqo]

This looks similar to one outlined in #8704. Could you check if changing the config to one below fixes it?

common:
  storage:
    backend: gcs
      gcs:
        service_account: >-
          ${GCS_SA_VAR_NAME}

(note the >- to fold the new lines from the property's value).

[zakris-s]

Yeah that is the issue I actually came from, and that solution does not work. Using '|' or '|-' to fold the lines also does not work for me.

[armandgrillet]

Can you try

service_account: ${GCS_SA_VAR_NAME}

instead?

[zakris-s]

Oddly enough when I deployed the HELM chart using argoCD the error went away. Although its important to note it only works if I provide the env var like this: (Note the '|' character is used for folding instead of ">-".

common:
  storage:
    backend: gcs
      gcs:
        service_account: |
          ${GCS_SA_VAR_NAME}