Refactored CWL-Airflow

1. Faster CWL parsing and validation
2. Simplified syntax for new DAG creation

#!/usr/bin/env python3
from cwl_airflow.extensions.cwldag import CWLDAG
dag = CWLDAG(workflow="my_workflow.cwl", dag_id="my_dag")

Update to the latest Airflow 1.10.10

1.1.12

Update to the latest Airflow 1.10.10

Latest cwltool and airflow

Test the latest cwltool==2.0.20200224214940 and airflow==1.10.9

To avoid frequent releases on every not critical bug fix, files prefixed with master will correspond to the latest build from the master branch. Other files are built at the time of release.

Bug fixes, update to the latest Airflow==1.10.6

Some new features

• Use Singularity instead of Docker (default: False). Update airflow.cfg with

[cwl]
singularity = True

• Force to use container (default: true). Update airflow.cfg with

[cwl]
use_container = True

Launching MacOs App from the console:

1. Set AIRFLOW_HOME environment variable to use a custom location for Airflow home directory
2. Run configuration script (do it every time you change AIRFLOW_HOME)
   cwl-airflow.app/Contents/MacOS/init
   Script will run airflow initdb, update airflow.cfg, copy DAGs and create required connections
3. Run scheduler
   cwl-airflow.app/Contents/MacOS/airflow scheduler
4. Run API server (use -h to see additional parameters)
   cwl-airflow.app/Contents/MacOS/apiserver

Launching MacOs App from a Finder:
NOTE: the default location for AIRFLOW_HOME will be used ~/airflow

1. Double click on cwl-airflow.app
   Script will create three services in ~/Library/LaunchAgents
com.biowardrobe.airflow-scheduler.plist
com.biowardrobe.airflow-webserver.plist
com.biowardrobe.airflow-apiserver.plist
2. Use Load and Unload buttons to start and stop services.
3. If cwl-airflow.app is moved to another location, make sure to manually remove abovementioned plist files

Launching portable Ubuntu 18.04 cwl-airflow version from the console:

1. Set AIRFLOW_HOME environment variable to use a custom location for Airflow home directory
2. Run scheduler
   cwl-airflow/bin_portable/airflow scheduler
3. Run API server (use -h to see additional parameters)
   cwl-airflow/bin_portable/apiserver

Add WES support

Launching MacOs App from the console:

1. Set AIRFLOW_HOME environment variable to use a custom location for Airflow home directory
2. Run configuration script (do it every time you change AIRFLOW_HOME)
   cwl-airflow.app/Contents/MacOS/init
   Script will run airflow initdb, update airflow.cfg, copy DAGs and create required connections
3. Run scheduler
   cwl-airflow.app/Contents/MacOS/airflow scheduler
4. Run API server (use -h to see additional parameters)
   cwl-airflow.app/Contents/MacOS/apiserver

Launching MacOs App from a Finder:
NOTE: the default location for AIRFLOW_HOME will be used ~/airflow

1. Double click on cwl-airflow.app
   Script will create three services in ~/Library/LaunchAgents
com.biowardrobe.airflow-scheduler.plist
com.biowardrobe.airflow-webserver.plist
com.biowardrobe.airflow-apiserver.plist
2. Use Load and Unload buttons to start and stop services.
3. If cwl-airflow.app is moved to another location, make sure to manually remove abovementioned plist files

Add scripts to run Airflow from the App

Launching MacOs App from the console:

1. Set AIRFLOW_HOME environment variable to use a custom location for Airflow home directory
2. Run configuration script (do it every time you change AIRFLOW_HOME)
   cwl-airflow.app/Contents/MacOS/init
   Script will run airflow initdb, update airflow.cfg, copy DAGs and create required connections
3. Run scheduler
   cwl-airflow.app/Contents/MacOS/airflow scheduler
4. Run API server (use -h to see additional parameters)
   cwl-airflow.app/Contents/MacOS/apiserver

Launching MacOs App from a Finder:
NOTE: the default location for AIRFLOW_HOME will be used ~/airflow

1. Double click on cwl-airflow.app
   Script will create three services in ~/Library/LaunchAgents
com.biowardrobe.airflow-scheduler.plist
com.biowardrobe.airflow-webserver.plist
com.biowardrobe.airflow-apiserver.plist
2. Use Load and Unload buttons to start and stop services.
3. If cwl-airflow.app is moved to another location, make sure to manually remove abovementioned plist files

Bug fix

1.1.8

Remove mysqlclient from dependencies, fix shebang bug

Bump cwltool version

Previous cwltool==1.0.20190621234233 had some bug with missing secondaryFiles

Bug fix, readme update

1.1.6

readme update

Bump version to be able to push it to Pypi

Identical to 1.1.4. Increment version to solve the problem of pushing release to Pypi