Dockerized Apache Airflow with Postgres Backend.

This Docker image contains Airflow 2.0 with minimal dependencies which makes the image lighter and more customizable.

All the docker-compose*.yml examples were developed taking into account all the best practices that are used for deploying docker containers in our company - Wizart Tech.

Prerequisites

• Install Docker
• Install Docker Compose
• Follow Apache Airflow Principles and How-To Guides in Docs

Supported Executors

• LocalExecutor
• CeleryExecutor

Security

1. In order to secure Airflow Connections and Variables, use fernet_key encryption. It can be done using Python:

   >> from cryptography.fernet import Fernet
   >> Fernet.generate_key().decode()
   '=69ksvvORDpeoBrz2N38El18kOxJFPU2peg22So66k7U=' # here is your fernet key

2. Specify the secret_key setting under the [webserver] config. Change this value to a new, per-environment, randomly generated string.

   For example using this command openssl rand -hex 30

Store the generated keys in env_file and name It airflow.env, just like It shown below:

AIRFLOW__CORE__FERNET_KEY=69ksvvORDpeoBrz2N38El18kOxJFPU2peg22So66k7U=
AIRFLOW__WEBSERVER__SECRET_KEY=1ca384d704f852756df25a7560c3338cb3a65cccf2fd734440f94deb5d32

Note: You can use airflow.env to define any container-level configurations for Airflow.

Additional information about securing connections can be found here.

Usage

For the first run, build containers using predefined make shortcuts:

make up

to list all available make shortcuts, type

make help

Create Users

docker-compose run --rm webserver bash  # or `make shell-root`
airflow users create \
    --role Admin \
    --username admin \
    --firstname FIRST_NAME \
    --lastname LAST_NAME \
    --email [email protected]

Executing airflow commands

If you want to run any of airflow commands, you can do the following: docker-compose run --rm webserver [some command]

• docker-compose run --rm webserver airflow dags list - List dags
• docker-compose run --rm webserver airflow tasks test [DAG_ID] [TASK_ID] [EXECUTION_DATE] - Test specific task
• docker-compose run --rm webserver python /usr/local/airflow/dags/[PYTHON-FILE].py - Test custom python script