-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change microbatch lookback
default from 0
to 1
#10876
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #10876 +/- ##
==========================================
- Coverage 89.20% 89.12% -0.08%
==========================================
Files 183 183
Lines 23464 23464
==========================================
- Hits 20930 20913 -17
- Misses 2534 2551 +17
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Changes to schemas.getdbt.com dbt-labs/schemas.getdbt.com#68 |
hey @QMalcolm , when this pr is merged, am i right in changing the docs here from |
That is correct 🙂 |
docs pr here to address this: https://github.com/dbt-labs/docs.getdbt.com/pull/6351/files |
Something weird happened. Some commits from another one of my branches got onto this one. I assume in my delirium of spinning plates, I did something dumb. |
02d44f4
to
bbb8e7e
Compare
bbb8e7e
to
8120810
Compare
hey @QMalcolm ! now that this is merged, am i ok merging the docs pr so the docs are updated? |
this pr adds updates to incremental microbatch per core prs: - [#10878](dbt-labs/dbt-core#10878) - makes it so --event-time-start adn --event-time-end are mutually required. - [#10876](dbt-labs/dbt-core#10876) - changes lookback default window to 1 (from 0) [ X ] dbt Core PRs must get merged first before docs pr is merged <!-- vercel-deployment-preview --> --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-update-microbatch-dbt-labs.vercel.app/docs/build/incremental-microbatch <!-- end-vercel-deployment-preview -->
Resolves #10867
Problem
A
lookback
value of0
results in microbatch models consistently having incomplete data.Solution
Change the
lookback
default to1
. This guarantees that batches will eventually get completed (assuming the job is run once perbatch_size
)Checklist