Skip to content

Commit

Permalink
Fix issue related to Airflow 2.4 & Werkzeug =>3 (#644)
Browse files Browse the repository at this point in the history
1) Airflow 2.4 & Werkzeug =>3 issue

Our integration tests failed for Airflow 2.4 and under since Werkzeug
3.0 and higher during `airflow db init` with:
```
Traceback (most recent call last):
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/bin/airflow", line 10, in <module>
    sys.exit(main())
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/airflow/__main__.py", line 39, in main
    args.func(args)
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/airflow/cli/cli_parser.py", line 52, in command
    return func(*args, **kwargs)
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/airflow/cli/commands/db_command.py", line 37, in initdb
    db.initdb()
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/airflow/utils/session.py", line 75, in wrapper
    return func(*args, session=session, **kwargs)
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/airflow/utils/db.py", line 681, in initdb
    _create_db_from_orm(session=session)
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/airflow/utils/db.py", line 653, in _create_db_from_orm
    from airflow.www.fab_security.sqla.models import Model
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/airflow/www/fab_security/sqla/models.py", line 27, in <module>
    from flask_appbuilder.models.sqla import Model
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/flask_appbuilder/__init__.py", line 5, in <module>
    from .api import ModelRestApi  # noqa: F401
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/flask_appbuilder/api/__init__.py", line 24, in <module>
    from ..baseviews import AbstractViewApi
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/flask_appbuilder/baseviews.py", line 22, in <module>
    from .forms import GeneralModelConverter
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/flask_appbuilder/forms.py", line 3, in <module>
    from flask_wtf import FlaskForm
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/flask_wtf/__init__.py", line 3, in <module>
    from .recaptcha import *
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/flask_wtf/recaptcha/__init__.py", line 2, in <module>
    from .fields import *
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/flask_wtf/recaptcha/fields.py", line 3, in <module>
    from . import widgets
  File "/Users/tati/Library/Application Support/hatch/env/virtual/astronomer-cosmos/4VBJdS-x/tests.py3.10-2.4/lib/python3.10/site-packages/flask_wtf/recaptcha/widgets.py", line 2, in <module>
    from werkzeug.urls import url_encode
```
Which led to no Airflow tables being created during the tests
initialisation.

2) Example DAG `cosmos_manifest_example` relying on pre-existing data

The selection statement used in the `cosmos_manifest_example` expected
data to pre-exist in the database. This probably worked before because
of the order of the tests run - and it broke up recently once this test
started running before others.

Since tests should be self-contained, the filter clause was changed so
this DAG execution becomes independent.

(cherry picked from commit 14b3090)
  • Loading branch information
tatiana committed Nov 6, 2023
1 parent 1a26591 commit 6e15291
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 2 deletions.
2 changes: 1 addition & 1 deletion dev/dags/cosmos_manifest_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
project_name="jaffle_shop",
),
profile_config=profile_config,
render_config=RenderConfig(load_method=LoadMode.DBT_MANIFEST, select=["path:models/customers.sql"]),
render_config=RenderConfig(load_method=LoadMode.DBT_MANIFEST, select=["path:seeds/raw_customers.csv"]),
operator_args={"install_deps": True},
# normal dag parameters
schedule_interval="@daily",
Expand Down
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,8 @@ dependencies = [
"types-attrs",
"types-requests",
"types-python-dateutil",
"apache-airflow"
"apache-airflow",
"Werkzeug<3.0.0",
]

[[tool.hatch.envs.tests.matrix]]
Expand Down

0 comments on commit 6e15291

Please sign in to comment.