https://github.com/noelmcloughlin/airflow-component
To assist sprint planning, this document is an exhaustive list of "ticket slogans" which to be refined as tasks for your infra team to be consumed by airflow-component installer or another installer, based on Official airflow docs:
primary: controller01.main.net user: main\airflowservice - Active Scheduler, UI, worker
secondary: controller02.main.net user: main\airflowservice - Standby Scheduler, UI, worker
worker01/02: apples, applesdev, applestest
worker01/02: oranges, orangesdev, orangestest
worker01/02: edge
worker01/02: fog
Request compute for apache-airflow controllers and workers. OS must be GNU/Linux (RedHat, Ubuntu):
-
[controller] Commission two [controllertype] compute hosts
-
[applesdev] Commision two [workertype] compute hosts
-
[orangesdev] Commision two [workertype] compute hosts
-
[applestest] Commision two [workertype] compute hosts
-
[orangestest] Commision two [workertype] compute hosts
-
[apples] Commision two [workertype] compute hosts
-
[oranges] Commision two [workertype] compute hosts
-
[edge] Commision two [workertype] compute hosts
-
[fog] Commision two [workertype] compute hosts
Request these entries be added to proxy for airflow-component Installer:
*.apache.org:443
*.bootstrap.pypa.io:443
*.postgresql.org:443
*.saltstack.com:443
*.fedoraproject.org:443
*.github.com:443
*.githubusercontent.com:443
*.fedora.is:443
*.cloudsmith.io:443
*.packagecloud.io
*.saltproject.io:443
*.linux.duke.edu :443
*.cloudfront.net:443
*.ftp.fi.muni.cz:443
*.mirrors.up.pt:443
*.mirrors.nxthost.com:443
*.fedora.cu.be:443
*.epel.mirror.omnilance.com:443
*.mirror.netzwerge.de:443
-
[controller] update proxy with [installer] mirrors
-
[applesdev] update proxy with [installer] mirrors
-
[orangesdev] update proxy with [installer] mirrors
-
[applestest] update proxy with [installer] mirrors
-
[orangestest] update proxy with [installer] mirrors
-
[apples] update proxy with [installer] mirrors
-
[oranges] update proxy with [installer] mirrors
-
[edge] update proxy with [installer] mirrors
-
[fog] update proxy with [installer] mirrors
Request access to airflow database from workers.
-
[applesdev] open access to remote postgres port 5432
-
[orangesdev] open access to remote postgres port 5432
-
[applestest] open access to remote postgres port 5432
-
[orangestest] open access to remote postgres port 5432
-
[apples] open access to remote postgres port 5432
-
[oranges] open access to remote postgres port 5432
-
[edge] open access to remote postgres port 5432
-
[fog] open access to remote postgres port 5432
Request access to worker task log files from controllers:
-
[applesdev] allow controllers to access worker port 8793 [airflow]
-
[orangesdev] allow controllers to access worker port 8793 [airflow]
-
[applestest] allow controllers to access worker port 8793 [airflow]
-
[orangestest] allow controllers to access worker port 8793 [airflow]
-
[apples] allow controllers to access worker port 8793 [airflow]
-
[oranges] allow controllers to access worker port 8793 [airflow]
-
[edge] allow controllers to access worker port 8793 [airflow]
-
[fog] allow controllers to access worker port 8793 [airflow]
Workers must be able to communicated with RabbitMQ on the controller domain per this ticket.
-
[applesdev] open access to [federated rabbitmq] port 5672
-
[orangesdev] open access to [federated rabbitmq] port 5672
-
[applestest] open access to [federated rabbitmq] port 5672
-
[orangestest] open access to [federated rabbitmq] port 5672
-
[apples] open access to [federated rabbitmq] port 5672
-
[oranges] open access to [federated rabbitmq] port 5672
-
[edge] open access to [federated rabbitmq] port 5672
-
[fog] open access to [federated rabbitmq] port 5672
This ticket allows PAIRS (controller01/02, worker01/02) to form cluster pairs (a<->b) over standard ports. We will create multiple independent rabbitmq two-node clusters for contoller, applesdev, applestest, apples, orangesdev, orangestest, oranges, edge, fog domains:
-
[controller] allow controller01/02 to form cluster on ports 25672,4369
-
[applesdev] allow worker01/02 to form cluster on ports 25672,4369
-
[orangesdev] allow worker01/02 to form cluster on ports 25672,4369
-
[applestest] allow worker01/02 to form cluster on ports 25672,4369
-
[orangestest] allow worker01/02 to form cluster on ports 25672,4369
-
[apples] allow worker01/02 to form cluster on ports 25672,4369
-
[oranges] allow worker01/02 to form cluster on ports 25672,4369
-
[edge] allow worker01/02 to form cluster on ports 25672,4369
-
[fog] allow worker01/02 to form cluster on ports 25672,4369
Allow synchronization of our git repo to controllers and workers (fog workers have no network connectivity).
-
[controller] allow git clone access to our repo
-
[applesdev] allow git clone access to our repo
-
[orangesdev] allow git clone access to our repo
-
[applestest] allow git clone access to our repo
-
[orangestest] allow git clone access to our repo
-
[apples] allow git clone access to our repo
-
[oranges] allow git clone access to our repo
-
[edge] allow git clone access to our repo
The installer works with POSIX Linux users. Maybe you need something else (i.e. "Service Accounts" for cloud resources):
-
[controller] create [airflow] airflowservice account
-
[applesdev] create [airflow] airflowservice account
-
[applestest] create [airflow] airflowservice account
-
[apples] create [airflow] airflowservice account
-
[orangesdev] create [airflow] airflowservice account
-
[orangestest] create [airflow] airflowservice account
-
[oranges] create [airflow] airflowservice account
-
[edge] create [airflow] airflowservice account