Switch from environment variable to behavior flag for gating microbatch functionality #323
Conversation
dbt-tests-adapter/dbt/tests/adapter/incremental/test_incremental_microbatch.py
Outdated
Show resolved
Hide resolved
QMalcolm
force-pushed
the
microbatch-behavior-flag
branch
from
da65029 to
88db9c8
Compare
|
So it turns out, we actually need to do MUCH less. I'm going to drop a bunch of commits... Like half of them. And then also squash some of them. And then make some changes to simplify the tests further. |
QMalcolm
force-pushed
the
microbatch-behavior-flag
branch
from
261e9ac to
89ca753
Compare
…stom_microbatch_strategy` The situation in which we want to warn the user about the behavior flag is exceedingly small. Specifically, we want to fire a deprecation warning when 1. The behavior flag is false 2. The user is executing a `microbatch` model 3. The user has a custome `microbatch` macro In dbt-adapters we could check (1) and (2), but we have no ability to check (3). Thus emitting the warning in dbt-adapters upon evaluation of the behavior flag `require_batched_execution_for_custom_microbatch_strategy` is _undesirable_. Instead we'll be emitting a deprecation warning from dbt-core.
This reverts commit 2b42247.
| @@ -63,7 +61,6 @@ def assert_row_count(self, project, relation_name: str, expected_row_count: int) | |||
|
|
|||
| assert len(result) == expected_row_count, f"{relation_name}:{pformat(result)}" | |||
|
|
|||
| @mock.patch.dict(os.environ, {"DBT_EXPERIMENTAL_MICROBATCH": "True"}) | |||
There was a problem hiding this comment.
Nice! Let's make sure to apply this change to concrete adapters, I think it's just BigQuery. The existing tests shouldn't break if we left it in, so it's just mostly for cleanliness:
(feel free to start a fresh dbt-bigquery PR if you'd like!)
|
Release planning: we'll need (patch?) releases of dbt-tests-adapter as well as Edit (thinking out loud more here)
What do you think? |
That feels right to me. @mikealfare Does that sound right? |
resolves #327
Problem
We're currently gating microbatch functionality via environment variables, and we should instead use a behavior flag
Solution
Switch to behavior flag for gating microbatch functionality
Checklist