Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix DbtRunOperationLocalOperator missing flags #529

Merged
merged 2 commits into from
Sep 14, 2023
Merged

Conversation

tatiana
Copy link
Collaborator

@tatiana tatiana commented Sep 12, 2023

@jensenity reported an issue in Slack related to Cosmos 1.1:
https://apache-airflow.slack.com/archives/C059CC42E9W/p1694488079896779

"for some reason, the the dbtrunoperation operator doesn’t use the cmd_flags….. it didnt have the --args flags when i look into my logs."

DbtRunOperationOperator(
                            task_id="run_operation",
                            macro_name="stage_external_sources",
                            args=dict(select='source_a'),
                            profile_config=profile_config,
                            project_dir=DBT_PROJECT_PATH,
                            install_deps=True,
                            env={
                                …
                            },
                            dbt_executable_path=DBT_EXECUTION_PATH,
                            queue="kubernetes",
                        )

This PR adds tests to all local operators to confirm if they are being executed with flags or not to avoid regressions.
It increases the project test coverage from 91.80% to 92.33%.

As a bonus, this PR also enables the --full-refresh flag when running dbt models, as requested by Monideep De in the Slack channel:
https://apache-airflow.slack.com/archives/C059CC42E9W/p1694523446939829

This change makes sense since this flag is supported both in Seeds and Models (https://docs.getdbt.com/reference/resource-configs/full_refresh), but we only exposed it in Seeds.

Bonus: add tests to all local operators to confirm if they are being executed with flags or not, to avoid regressions.
@tatiana tatiana requested a review from a team as a code owner September 12, 2023 10:24
@tatiana tatiana requested a review from a team September 12, 2023 10:24
@netlify
Copy link

netlify bot commented Sep 12, 2023

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name Link
🔨 Latest commit fe36958
🔍 Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/65006c5e8b4bd0000897f10d

@tatiana tatiana changed the title Add flags to DbtRunOperationLocalOperator Fix missing flags on DbtRunOperationLocalOperator Sep 12, 2023
@tatiana tatiana temporarily deployed to internal September 12, 2023 10:24 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 12, 2023 10:49 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Sep 12, 2023

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.56% 🎉

Comparison is base (f2dd7e1) 91.80% compared to head (fe36958) 92.37%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #529      +/-   ##
==========================================
+ Coverage   91.80%   92.37%   +0.56%     
==========================================
  Files          49       49              
  Lines        1915     1927      +12     
==========================================
+ Hits         1758     1780      +22     
+ Misses        157      147      -10     
Files Changed Coverage Δ
cosmos/operators/local.py 88.15% <100.00%> (+4.15%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tatiana tatiana changed the title Fix missing flags on DbtRunOperationLocalOperator Fix DbtRunOperationLocalOperator missing flags Sep 12, 2023
@tatiana tatiana temporarily deployed to internal September 12, 2023 13:49 — with GitHub Actions Inactive
@tatiana tatiana added this to the 1.1.1 milestone Sep 12, 2023
@tatiana tatiana temporarily deployed to internal September 13, 2023 09:01 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 13, 2023 09:42 — with GitHub Actions Inactive
Copy link
Contributor

@harels harels left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@tatiana tatiana merged commit bcf7714 into main Sep 14, 2023
36 checks passed
@tatiana tatiana deleted the run-operation-operator branch September 14, 2023 10:22
tatiana added a commit that referenced this pull request Sep 14, 2023
Bug fixes

* Fix attempt of emitting OpenLineage events if task execution fails by @tatiana in #526
* Fix Rust dependency for Windows users by @tatiana in #526
* Fix DbtRunOperationLocalOperator missing flags by @tatiana in #529
* Fix DbtRunLocalOperator to support the full refresh argument by @tatiana in #529
* Remove redundant prefix of task names when test_behavior = TestBehavior.AFTER_EACH by @binhnq94 in #524
* Fix rendering vars in `DbtModel` when using `LoadMode.CUSTOM` by @dojinkimm in #502

Others

* Docs: add `documentation comparing Airflow and dbt concepts <https://astronomer.github.io/astronomer-cosmos/getting_started/dbt-airflow-concepts.html>`_ by @tatiana in #523.
* Update PyPI project links by @tatiana in #528
* pre-commit updates
tatiana added a commit that referenced this pull request Sep 14, 2023
Bug fixes

* Fix attempt of emitting OpenLineage events if task execution fails by @tatiana in #526
* Fix Rust dependency for Windows users by @tatiana in #526
* Fix DbtRunOperationLocalOperator missing flags by @tatiana in #529
* Fix DbtRunLocalOperator to support the full refresh argument by @tatiana in #529
* Remove redundant prefix of task names when test_behavior = TestBehavior.AFTER_EACH by @binhnq94 in #524
* Fix rendering vars in `DbtModel` when using `LoadMode.CUSTOM` by @dojinkimm in #502

Others

* Docs: add `documentation comparing Airflow and dbt concepts <https://astronomer.github.io/astronomer-cosmos/getting_started/dbt-airflow-concepts.html>`_ by @tatiana in #523.
* Update PyPI project links by @tatiana in #528
* pre-commit updates
@tatiana tatiana mentioned this pull request Sep 14, 2023
harels pushed a commit that referenced this pull request Sep 14, 2023
Bug fixes

* Remove redundant prefix of task names when test_behavior =
TestBehavior.AFTER_EACH by @binhnq94 in #524
* Fix rendering vars in `DbtModel` when using `LoadMode.CUSTOM` by
@dojinkimm in #502
* Fix attempt of emitting OpenLineage events if task execution fails by
@tatiana in #526
* Fix Rust dependency for Windows users by @tatiana in #526
* Fix DbtRunOperationLocalOperator missing flags by @tatiana in #529
* Fix DbtRunLocalOperator to support the full refresh argument by
@tatiana in #529

Others

* Docs: add `documentation comparing Airflow and dbt concepts
<https://astronomer.github.io/astronomer-cosmos/getting_started/dbt-airflow-concepts.html>`_
by @tatiana in #523.
* Update PyPI project links by @tatiana in #528
* pre-commit updates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants