Unusable airflow.variables and airflow.extraVar

[JulienDelavande]

Checks

• I have checked for existing issues.
• This report is about the User-Community Airflow Helm Chart.

Chart Version

8.9.0

Kubernetes Version

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.4+k3s1

Helm Version

version.BuildInfo{Version:"v3.16.0-rc.1", GitCommit:"0d439e1a09683f21a0ab9401eb661401f185b00b", GitTreeState:"clean", GoVersion:"go1.22.6"}

Description

Title: airflow.variables and extraEnv not applied in env and envFrom sections from values.yaml

Description:

According to the documentation, it should be possible to define environment variables and other settings in the values.yaml file through the airflow.variables and airflow.extraEnv fields. However, these fields do not seem to be used or referenced within the Helm templates under the env and envFrom sections for Airflow pods.

For example, in values.yaml:

airflow:
  variables:
    - key: "VAR_TEST_KEY"
      value: "VAR_TEST_VALUE"
  extraEnv:
    - ConfigMapKeySelector: "global-config"

Expected Behavior:

• The variables defined in airflow.variables should be loaded as environment variables in the pods.
• The airflow.extraEnv field should allow the user to specify extra environment variables using ConfigMapKeySelector or SecretKeySelector.

For example:

airflow:
  extraEnv:
    - name: CUSTOM_VAR
      valueFrom:
        configMapKeyRef:
          name: global-config
          key: CUSTOM_VAR

These settings should then be applied under the env and envFrom sections of the Airflow deployment templates.

Actual Behavior:

Currently, the fields airflow.variables and airflow.extraEnv are not referenced or applied in the env or envFrom sections of the webserver, scheduler, or worker pods.

Steps to Reproduce:

1. Define variables and extra environment settings in values.yaml as follows:

airflow:
  variables:
    - key: "VAR_TEST_KEY"
      value: "VAR_TEST_VALUE"
  extraEnv:
    - name: CUSTOM_VAR
      valueFrom:
        configMapKeyRef:
          name: global-config
          key: CUSTOM_VAR

2. Deploy the chart with Helm:

helm upgrade --install my-airflow-release apache-airflow/airflow -f values.yaml

3. Check the environment variables in the Airflow pods:

kubectl describe pod <pod-name>

The expected variables (VAR_TEST_KEY and CUSTOM_VAR) will not be present in the pod's environment.

Proposed Solution:

Update the Helm chart templates to properly reference and apply airflow.variables and airflow.extraEnv fields in the env and envFrom sections. This will ensure that variables and extra environment configurations specified in values.yaml are correctly injected into the Airflow pods.

For example, the webserver deployment template could be updated as follows:

containers:
  - name: airflow-web
    env:
      {{- include "airflow.env" . | indent 8 }}
      {{- toYaml .Values.airflow.variables | indent 8 }}
    envFrom:
      {{- toYaml .Values.airflow.extraEnv | indent 8 }}

This will allow for the proper inclusion of environment variables defined in airflow.variables and extraEnv.

Relevant Logs

kubectl exec -it optimsportbets-web-69f9c7d8d5-4ntr5 -- env | grep VAR_TEST_KEY

output: None

Custom Helm Values

airflow:
  variables:
    - key: "VAR_TEST_KEY"
      value: "VAR_TEST_VALUE"
  extraEnv:
    - name: CUSTOM_VAR
      valueFrom:
        configMapKeyRef:
          name: global-config
          key: CUSTOM_VAR