Implement prefix + DBT selector graph operator for model selection #429
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #429 +/- ##
==========================================
+ Coverage 93.13% 93.16% +0.03%
==========================================
Files 51 51
Lines 2053 2079 +26
==========================================
+ Hits 1912 1937 +25
- Misses 141 142 +1
☔ View full report in Codecov by Sentry. |
tseruga
force-pushed
the
custom-selector-upstream-model
branch
from
8f6f778 to
85499f1
Compare
|
Added another test to hopefully make Coverage happy -- and rebased off of main. |
|
@tseruga, this work was looking really good. It would be great if we could rebase it and get to the finishing line! Please let us know if you need any support. |
|
Sure thing. This got dropped for other priorities, but I can spend some time in the coming days/weeks to try to wrap this up |
tseruga
force-pushed
the
custom-selector-upstream-model
branch
from
f40629b to
74d1b8f
Compare
|
Thanks a lot, @tseruga ! Please let us know when the PR is ready for review - I've assigned it for the 1.3 release, and we can review it if necessary. |
tatiana
added
the
area:selector
tatiana
added a commit
that referenced
this pull request
Dec 5, 2023
…OM` or `LoadMode.DBT_MANIFEST` (#728) Add support for the following when using `LoadMode.CUSTOM` or `LoadMode.DBT_MANIFEST`: * Support selection of model by name * Support the selection of models by name & their children (with or without degrees) * Support the selection of models by name & their parents (with or without degrees) * Support intersections and unions involving graph selectors (with or without other supported selectors, eg. tags) Examples of select/exclusion statements that now work regardless of the `LoadMode` being used: ``` model_a +model_b model_c+ +model_d+ 2+model_e model_f+3 model_f+,tag:nightly ``` Related dbt documentation: https://docs.getdbt.com/reference/node-selection/graph-operators https://docs.getdbt.com/reference/node-selection/set-operators Limitations: * The at operator is not supported yet (`@`) * If users opt to use graph selector, it will increase the DAG parsing time and the task execution time when using `LoadMode.CUSTOM` or `LoadMode.DBT_MANIFEST` This PR improves and extends the original implementation proposed by @tseruga in #429. Some of the changes that were introduced on top of the original PR: * Add support to descendants (before only precursors were supported) * Add support to different depths/degrees of precursors/descendants * Add support to the union between graph operators and graph/non-graph operators * Add support to the intersection between graph operators and graph/non-graph operators Closes: #684 Co-authored-by: Tyler Seruga <[email protected]>
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
…OM` or `LoadMode.DBT_MANIFEST` (astronomer#728) Add support for the following when using `LoadMode.CUSTOM` or `LoadMode.DBT_MANIFEST`: * Support selection of model by name * Support the selection of models by name & their children (with or without degrees) * Support the selection of models by name & their parents (with or without degrees) * Support intersections and unions involving graph selectors (with or without other supported selectors, eg. tags) Examples of select/exclusion statements that now work regardless of the `LoadMode` being used: ``` model_a +model_b model_c+ +model_d+ 2+model_e model_f+3 model_f+,tag:nightly ``` Related dbt documentation: https://docs.getdbt.com/reference/node-selection/graph-operators https://docs.getdbt.com/reference/node-selection/set-operators Limitations: * The at operator is not supported yet (`@`) * If users opt to use graph selector, it will increase the DAG parsing time and the task execution time when using `LoadMode.CUSTOM` or `LoadMode.DBT_MANIFEST` This PR improves and extends the original implementation proposed by @tseruga in astronomer#429. Some of the changes that were introduced on top of the original PR: * Add support to descendants (before only precursors were supported) * Add support to different depths/degrees of precursors/descendants * Add support to the union between graph operators and graph/non-graph operators * Add support to the intersection between graph operators and graph/non-graph operators Closes: astronomer#684 Co-authored-by: Tyler Seruga <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a commonly used graph operator for model selection in DBT.
https://docs.getdbt.com/reference/node-selection/graph-operators#the-plus-operator
This is just a first stab at implementing (what I assume is) the most commonly used graph operator. It just so happens that it's also the easiest to implement.
This change allows users to specify the DBT model they wish to select, as well as its upstream dependencies.
select=["+my_model"]will return themy_modelnode as well as any of the nodes that itdepends_on, and any nodes those nodes depend on, etc.Haven't written up the docs because I want to make sure folks are happy with handling the graph operators this way. It might limit future implementation of the remaining operators, so happy to hear feedback on the topic.
This is just what I hacked together while trying to play around with Cosmos post-1.0.0 :)
NOTE: I also cherry-picked #426 in order to get the tests to work, but will rebase when required.
Checklist