-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix DbtRunOperationLocalOperator missing flags #529
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Bonus: add tests to all local operators to confirm if they are being executed with flags or not, to avoid regressions.
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
tatiana
changed the title
Add flags to DbtRunOperationLocalOperator
Fix missing flags on DbtRunOperationLocalOperator
Sep 12, 2023
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #529 +/- ##
==========================================
+ Coverage 91.80% 92.37% +0.56%
==========================================
Files 49 49
Lines 1915 1927 +12
==========================================
+ Hits 1758 1780 +22
+ Misses 157 147 -10
☔ View full report in Codecov by Sentry. |
tatiana
changed the title
Fix missing flags on DbtRunOperationLocalOperator
Fix DbtRunOperationLocalOperator missing flags
Sep 12, 2023
jensenity
reviewed
Sep 13, 2023
harels
approved these changes
Sep 13, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice!
tatiana
added a commit
that referenced
this pull request
Sep 14, 2023
Bug fixes * Fix attempt of emitting OpenLineage events if task execution fails by @tatiana in #526 * Fix Rust dependency for Windows users by @tatiana in #526 * Fix DbtRunOperationLocalOperator missing flags by @tatiana in #529 * Fix DbtRunLocalOperator to support the full refresh argument by @tatiana in #529 * Remove redundant prefix of task names when test_behavior = TestBehavior.AFTER_EACH by @binhnq94 in #524 * Fix rendering vars in `DbtModel` when using `LoadMode.CUSTOM` by @dojinkimm in #502 Others * Docs: add `documentation comparing Airflow and dbt concepts <https://astronomer.github.io/astronomer-cosmos/getting_started/dbt-airflow-concepts.html>`_ by @tatiana in #523. * Update PyPI project links by @tatiana in #528 * pre-commit updates
tatiana
added a commit
that referenced
this pull request
Sep 14, 2023
Bug fixes * Fix attempt of emitting OpenLineage events if task execution fails by @tatiana in #526 * Fix Rust dependency for Windows users by @tatiana in #526 * Fix DbtRunOperationLocalOperator missing flags by @tatiana in #529 * Fix DbtRunLocalOperator to support the full refresh argument by @tatiana in #529 * Remove redundant prefix of task names when test_behavior = TestBehavior.AFTER_EACH by @binhnq94 in #524 * Fix rendering vars in `DbtModel` when using `LoadMode.CUSTOM` by @dojinkimm in #502 Others * Docs: add `documentation comparing Airflow and dbt concepts <https://astronomer.github.io/astronomer-cosmos/getting_started/dbt-airflow-concepts.html>`_ by @tatiana in #523. * Update PyPI project links by @tatiana in #528 * pre-commit updates
Merged
harels
pushed a commit
that referenced
this pull request
Sep 14, 2023
Bug fixes * Remove redundant prefix of task names when test_behavior = TestBehavior.AFTER_EACH by @binhnq94 in #524 * Fix rendering vars in `DbtModel` when using `LoadMode.CUSTOM` by @dojinkimm in #502 * Fix attempt of emitting OpenLineage events if task execution fails by @tatiana in #526 * Fix Rust dependency for Windows users by @tatiana in #526 * Fix DbtRunOperationLocalOperator missing flags by @tatiana in #529 * Fix DbtRunLocalOperator to support the full refresh argument by @tatiana in #529 Others * Docs: add `documentation comparing Airflow and dbt concepts <https://astronomer.github.io/astronomer-cosmos/getting_started/dbt-airflow-concepts.html>`_ by @tatiana in #523. * Update PyPI project links by @tatiana in #528 * pre-commit updates
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@jensenity reported an issue in Slack related to Cosmos 1.1:
https://apache-airflow.slack.com/archives/C059CC42E9W/p1694488079896779
"for some reason, the the dbtrunoperation operator doesn’t use the cmd_flags….. it didnt have the --args flags when i look into my logs."
This PR adds tests to all local operators to confirm if they are being executed with flags or not to avoid regressions.
It increases the project test coverage from 91.80% to 92.33%.
As a bonus, this PR also enables the
--full-refresh
flag when running dbt models, as requested by Monideep De in the Slack channel:https://apache-airflow.slack.com/archives/C059CC42E9W/p1694523446939829
This change makes sense since this flag is supported both in Seeds and Models (https://docs.getdbt.com/reference/resource-configs/full_refresh), but we only exposed it in Seeds.