Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace environment variable with a project flag to gate microbatch functionality #10799

Merged
merged 28 commits into from
Nov 11, 2024
+754 −637
Merged

Replace environment variable with a project flag to gate microbatch functionality #10799

Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
a9df50f
first pass: replace os env with project flag
MichelleArk Sep 30, 2024
96c4ccb
Merge branch 'main' into microbatch-project-flags
QMalcolm Oct 1, 2024
704494f
Fix `TestMicrobatchMultipleRetries` to not use `os.env`
QMalcolm Oct 1, 2024
bfecde1
Turn off microbatch project flag for `TestMicrobatchCustomUserStrateg…
QMalcolm Oct 1, 2024
d690dec
Update `BaseMicrobatchTest` to turn on microbatch via project flags
QMalcolm Oct 1, 2024
5b9d7f7
[REVERT BEFORE MERGE] Update dbt-adapters dev req to point to branch …
QMalcolm Oct 1, 2024
1105e42
Add changie doc
QMalcolm Oct 1, 2024
a3f2708
Merge branch 'main' into microbatch-project-flags
QMalcolm Oct 15, 2024
18d1938
Merge branch 'main' into microbatch-project-flags
QMalcolm Nov 1, 2024
4d908ad
Fix functional tests after merging in main
QMalcolm Nov 4, 2024
0349968
Extract most of the logic from `find_materialization_macro_by_name` t…
QMalcolm Nov 5, 2024
a620919
Add function to that determines whether the new microbatch functional…
QMalcolm Nov 5, 2024
e77a13c
Gate microbatch functionality by `use_microbatch_batches` manifest fu…
QMalcolm Nov 5, 2024
543e024
Rename microbatch behavior flag to `require_batched_execution_for_cus…
QMalcolm Nov 5, 2024
db3fba8
Merge branch 'main' into microbatch-project-flags
QMalcolm Nov 6, 2024
c5e314a
Extract logic of `find_macro_by_name` to `find_macro_candiate_by_name`
QMalcolm Nov 7, 2024
35b23a3
Use `find_macro_candidate_by_name` to find the microbatch macro
QMalcolm Nov 7, 2024
7808992
Revert "Extract most of the logic from `find_materialization_macro_by…
QMalcolm Nov 7, 2024
7f88f96
Fix microbatch macro locality check to search for `core` locality ins…
QMalcolm Nov 7, 2024
874d7c6
Merge branch 'main' into microbatch-project-flags
QMalcolm Nov 8, 2024
a24b6f4
Move the evaluation of `use_microbatch_batches` to the last position …
QMalcolm Nov 8, 2024
dff25de
Drop behavior flag setting for BaseMicrobatchTest tests
QMalcolm Nov 8, 2024
3931b69
Rename 'env_var' to 'project_flag' in test_microbatch.py
QMalcolm Nov 8, 2024
945bcff
Update microbatch tests to assert when we are/aren't running with bat…
QMalcolm Nov 8, 2024
63445b5
Update `test_resolve_event_time_filter` to use `use_microbatch_batches`
QMalcolm Nov 8, 2024
26ba807
Fire deprecation warning for custom microbatch macros
QMalcolm Nov 8, 2024
94363e8
Add microbatch deprecation events to test_events.py
QMalcolm Nov 8, 2024
0f6b7fb
Revert "[REVERT BEFORE MERGE] Update dbt-adapters dev req to point to…
QMalcolm Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions dev-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
git+https://github.com/dbt-labs/dbt-adapters.git@main
git+https://github.com/dbt-labs/dbt-adapters.git@main#subdirectory=dbt-tests-adapter
git+https://github.com/dbt-labs/dbt-adapters.git@microbatch-behavior-flag
git+https://github.com/dbt-labs/dbt-adapters.git@microbatch-behavior-flag#subdirectory=dbt-tests-adapter
git+https://github.com/dbt-labs/dbt-common.git@main
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reminder: let's make sure we bump the lower bound pin of dbt-adapters in setup.py as part of this PR once the dbt-adapters branches are merged.

git+https://github.com/dbt-labs/dbt-postgres.git@main
# black must match what's in .pre-commit-config.yaml to be sure local env matches CI
Expand Down
Loading