Remote logging to Azure blob storage

[HakarK]

I am getting this response when I try to read the logs

I have followed this guide for setup

I have created an Airflow connection called my_wasb which works (or atleast I am getting "Successfully connected to Azure Blob Storage." when i test the connection)

And i have a storage account called airflowstvg and a container called wasb-airflowlogs.

The config section in values file looks like this

Do I need to follow these steps also?

[thesuperzapper]

@HakarK I think something must have changed in the Azure provider.

Before we try using an airflow_local_settings.py (which the chart provides values for), can you try formatting AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER like wasb://<container_name>@<storage_account>.blob.core.windows.net to see if that works?

Because right now, it looks like Airflow is not even trying to use a remote provider (probably because of the missing wasb:// scheme).

---

If that fixes it, we should make an update to both the chart's docs and the upstream airflow provider docs:

• https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/log-persistence.md
• https://github.com/apache/airflow/blob/main/docs/apache-airflow-providers-microsoft-azure/logging/index.rst

[HakarK]

I still get the same response as above when trying to follow the scheme wasb://<container_name>@<storage_account>.blob.core.windows.net

My storage account is named airflowstvg and container name wasb-airflowlogs:

[thesuperzapper]

@HakarK after reading the airflow implementation, I can confirm that Azure Blob Store logging is really messed up, as explained here.

Effectively, the problems are:

1. the Azure Blob Store container name has been hard-coded to airflow-logs
2. the REMOTE_BASE_LOG_FOLDER must start with wasb for airflow to know you are using Azure Blob Store (which causes blob names to always have a prefix that starts with wasb/)

However, we can work around these issues to get logs stored in the Azure Blob Store container named MY_CONTAINER, with the prefix CUSTOM_PREFIX/.

SOLUTION: Airflow 2.6

If you are using airflow 2.6, you can set:

• AIRFLOW__LOGGING__REMOTE_LOGGING to True
• AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID to "my_wasb" (which must have "login" set to your storage account name)
• AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER to "wasb" (which is only to tell airflow we are using Azure Blob Store)
• AIRFLOW__LOGGING__REMOTE_TASK_HANDLER_KWARGS to {"wasb_container": "MY_CONTAINER", "wasb_log_folder": "CUSTOM_PREFIX"}

SOLUTION: Airflow 2.0 - 2.5

If you are using airflow 2.0 - 2.5, you can set:

• AIRFLOW__LOGGING__REMOTE_LOGGING to True
• AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID to "my_wasb" (which must have "login" set to your storage account name)
• AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER to "wasb" (which is only to tell airflow we are using Azure Blob Store)
• AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS to "airflow_local_settings.LOGGING_CONFIG"

Then, you must create the following airflow_local_settings.py using the airflow.localSettings.stringOverride value:

airflow:
  localSettings:
    stringOverride: |
      from copy import deepcopy
      from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG

      LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)

      LOGGING_CONFIG["handlers"]["task"]["wasb_container"] = "MY_CONTAINER"
      LOGGING_CONFIG["handlers"]["task"]["wasb_log_folder"] = "CUSTOM_PREFIX"

Finally, you may need to make airflow see /opt/airflow/config/ as an importable package:

If you run into an import error for airflow_local_settings.LOGGING_CONFIG you will need to mount an empty __init__.py file into the /opt/airflow/config/ folder. You can do this with the airflow.extraVolumeMounts and airflow.extraVolumes values.

[HakarK]

With these values using airflow version 2.6.0:

login is set to storage account name (airflowstvg) in my_wasb connection.

The pods goes into init:CrashLoopBackOff status.
I have not seen any import error.

Is the value of wasb_log_folder arbitrary? or does it actually need to be the name of the folder inside the container (wasb-airflowlogs)? I made the folder logs in the container, but it still did not work.

[thesuperzapper]

@HakarK I think you may have confused my comment, I have updated it to be more clear. Note, for Airflow 2.6, you ONLY need to set the configs, not the local_settings.py.

Because I don't have an Azure ready to test, I would greatly appreciate if you can test two things:

1. Confirm that the "SOLUTION: Airflow 2.6" listed above works
2. Try using "SOLUTION: Airflow 2.0 - 2.5" (in airflow 2.6 is fine), and confirm if it works (and if the __init__.py mount is needed)

---

In both solutions, wasb_log_folder is the "file prefix" under which the logs will be written.

[HakarK]

Solution 1 did not work for me.
Solution 2 works, and I did not have to make the __init__.py file. I tested solution 2 with airflow version 2.4.3-python3.8

The documentation should be updated. Thank you for the help!

[thesuperzapper]

@HakarK just to clarify, when you tried solution 1 were you using Airflow 2.6?

Because solution 1 will only work with Airflow 2.6+, but solution 2 should work for all versions (including 2.6).

[HakarK]

I tried solution 1 with version 2.6.0 and 2.6.3.

[HakarK]

@thesuperzapper Is there documentation for reading DAGs from storage account?