Feature/decouple adapters from core #8906

MichelleArk · 2023-10-25T18:59:04Z

resolves #8917

Problem

Solution

Checklist

I have read the contributing guide and understand what's expected of me
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
This PR includes type annotations for new and modified functions

* Move events to common * More Type Annotations (#8536) * Extend use of type annotations in the events module. * Add return type of None to more __init__ definitions. * Still more type annotations adding -> None to __init__ * Tweak per review * Allow adapters to include python package logging in dbt logs (#8643) * add set_package_log_level functionality * set package handler * set package handler * add logging about stting up logging * test event log handler * add event log handler * add event log level * rename package and add unit tests * revert logfile config change * cleanup and add code comments * add changie * swap function for dict * add additional unit tests * fix unit test * update README and protos * fix formatting * update precommit --------- Co-authored-by: Peter Webb <[email protected]>

* moving types_pb2.py to common/events * split out utils into core/common/adapters * add changie

* remove usage of dbt.config.PartialProject from dbt/adapters * add changie --------- Co-authored-by: Colin <[email protected]>

…ters-from-core

* move agate_helper into common * add changie --------- Co-authored-by: Colin <[email protected]>

* moving types_pb2.py to common/events * Refactor Base Exceptions * update make_log_dir_if_missing to handle str * move remaining adapters exception imports to common/adapters --------- Co-authored-by: Michelle Ark <[email protected]>

…er-specific (#9051)

* Move constraints to dbt.common * Move constraints to contracts folder, per review * Add a changelog entry.

* moving types_pb2.py to common/events * Move AdapterLogger to adapter folder * add changie

* Move the semver package to common and alter references. * Alter leftover references to dbt.semver, this time using from syntax. --------- Co-authored-by: Mila Page <[email protected]>

* moving types_pb2.py to common/events * add config field to RelationConfig

colin-rogers-dbt · 2023-12-19T17:17:05Z

core/dbt/adapters/protocol.py

@@ -93,11 +89,26 @@ class AdapterProtocol(  # type: ignore[misc]
    def __init__(self, config: AdapterRequiredConfig) -> None:
        ...


Look at macro context args

colin-rogers-dbt · 2023-12-19T17:19:38Z

core/dbt/common/exceptions/base.py

@@ -0,0 +1,275 @@
+import builtins


Look at split of exceptions into separate files

colin-rogers-dbt · 2023-12-19T17:40:50Z

core/dbt/common/invocation.py

@@ -0,0 +1,12 @@
+import uuid
+
+_INVOCATION_ID = str(uuid.uuid4())


new invocation_id global

colin-rogers-dbt · 2023-12-19T23:41:44Z

core/dbt/adapters/contracts/macros.py

+from dbt.common.clients.jinja import MacroProtocol
+
+
+class MacroResolverProtocol(Protocol):


replaces referencing the manifest

colin-rogers-dbt · 2023-12-19T23:45:19Z

core/dbt/common/events/event_manager_client.py

+_EVENT_MANAGER: IEventManager = EventManager()
+
+
+def get_event_manager() -> IEventManager:


this functionality was moved from functions.py to make it clearer how the event manager is setup and managed

colin-rogers-dbt · 2023-12-19T23:48:49Z

core/dbt/common/events/__init__.py

+from dbt.common.events.logger import LineFormat
+
+# make sure event manager starts with a logger
+get_event_manager().add_logger(


this behavior was in factory.py, we have moved it to init as this is a more idiomatic place to locate behavior that we expect to happen on module import

* moving types_pb2.py to common/events * Update parser to support conversion metrics (#9173) * added ConversionTypeParams classes * updated parser for ConversionTypeParams * added step to populate input_measure for conversion metrics * version bump on DSI * comment back manifest generating line * updated v12 schemas * added tests * added changelog * Remove `--dry-run` flag from `dbt deps` (#9169) * Rm --dry-run flag for dbt deps * Add changelog entry * Update test * PR feedback * adding clean_up methods to basic and unique_id tests (#9195) * init attempt of adding clean_up methods to basic and unique_id tests * swapping cleanup method drop of test_schema to unique_schema to test breakage on docs_generate test * moving the clean_up method down into class BaseDocsGenerate * remove drop relation for unique_schema * manually define alternate_schema for clean_up as not being seen as part of project_config * add changelog * remove unneeded changelog * uncomment line that generates new manifest and delete manifest our changes created * make sure the manifest test is deleted and readd older version of manifest.json to appease test * manually revert file to previous commit * Revert "manually revert file to previous commit" This reverts commit a755419. --------- Co-authored-by: William Deng <[email protected]> Co-authored-by: Jeremy Cohen <[email protected]> Co-authored-by: Matthew McKnight <[email protected]>

ChenyuLInx

Thanks for getting the change up!! This looks mostly good to me!
Have we tried this branch with other 1-rst party adapters to get some rough understanding of how much work one adapter need to use this change?

ChenyuLInx · 2023-12-21T21:41:09Z

.changes/unreleased/Breaking Changes-20231129-091921.yaml

@@ -0,0 +1,7 @@
+kind: Breaking Changes
+body: Rm --dry-run flag from 'dbt deps --add-package', in favor of just 'dbt deps


This change seems not so relevant to what you are doing in this PR?

yea I'm not sure why this shows as a diff as the PR for this was merged last week?

ChenyuLInx · 2023-12-21T21:41:51Z

.changes/unreleased/Features-20231107-135635.yaml

@@ -0,0 +1,6 @@
+kind: Features
+body: Remove legacy logger


I think we don't use this on cloud anymore, do we need to double check?

@MichelleArk checked and they have all moved to event logging

colin-rogers-dbt · 2023-12-21T22:27:42Z

Thanks for getting the change up!! This looks mostly good to me! Have we tried this branch with other 1-rst party adapters to get some rough understanding of how much work one adapter need to use this change?

@ChenyuLInx yup I have a working PR up for bigquery: dbt-labs/dbt-bigquery#1026

* moving types_pb2.py to common/events * Update parser to support conversion metrics (#9173) * added ConversionTypeParams classes * updated parser for ConversionTypeParams * added step to populate input_measure for conversion metrics * version bump on DSI * comment back manifest generating line * updated v12 schemas * added tests * added changelog * Remove `--dry-run` flag from `dbt deps` (#9169) * Rm --dry-run flag for dbt deps * Add changelog entry * Update test * PR feedback * adding clean_up methods to basic and unique_id tests (#9195) * init attempt of adding clean_up methods to basic and unique_id tests * swapping cleanup method drop of test_schema to unique_schema to test breakage on docs_generate test * moving the clean_up method down into class BaseDocsGenerate * remove drop relation for unique_schema * manually define alternate_schema for clean_up as not being seen as part of project_config * add changelog * remove unneeded changelog * uncomment line that generates new manifest and delete manifest our changes created * make sure the manifest test is deleted and readd older version of manifest.json to appease test * manually revert file to previous commit * Revert "manually revert file to previous commit" This reverts commit a755419. --------- Co-authored-by: William Deng <[email protected]> Co-authored-by: Jeremy Cohen <[email protected]> Co-authored-by: Matthew McKnight <[email protected]>

* Update parser to support conversion metrics (#9173) * added ConversionTypeParams classes * updated parser for ConversionTypeParams * added step to populate input_measure for conversion metrics * version bump on DSI * comment back manifest generating line * updated v12 schemas * added tests * added changelog * Remove `--dry-run` flag from `dbt deps` (#9169) * Rm --dry-run flag for dbt deps * Add changelog entry * Update test * PR feedback * adding clean_up methods to basic and unique_id tests (#9195) * init attempt of adding clean_up methods to basic and unique_id tests * swapping cleanup method drop of test_schema to unique_schema to test breakage on docs_generate test * moving the clean_up method down into class BaseDocsGenerate * remove drop relation for unique_schema * manually define alternate_schema for clean_up as not being seen as part of project_config * add changelog * remove unneeded changelog * uncomment line that generates new manifest and delete manifest our changes created * make sure the manifest test is deleted and readd older version of manifest.json to appease test * manually revert file to previous commit * Revert "manually revert file to previous commit" This reverts commit a755419. --------- Co-authored-by: William Deng <[email protected]> Co-authored-by: Jeremy Cohen <[email protected]> Co-authored-by: Matthew McKnight <[email protected]>

## What are you changing in this pull request and why?  This PR updates various links after a [PR in dbt Core](dbt-labs/dbt-core#8906) moved some files. Additionally, it updates some of the links to point to the correct line of code. ## Checklist  - [x] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] For [docs versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning), review how to [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content). - [ ] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." Adding or removing pages (delete if not applicable): - [ ] Add/remove page in `website/sidebars.js` - [ ] Provide a unique filename for new pages - [ ] Add an entry for deleted pages in `website/static/_redirects` - [ ] Run link testing locally with `npm run build` to update the links that point to deleted pages Co-authored-by: mirnawong1 <[email protected]>

MichelleArk and others added 2 commits October 25, 2023 14:56

remove dbt.contracts.connection imports from adapter module

469a9ac

cla-bot bot added the cla:yes label Oct 25, 2023

This comment was marked as outdated.

Sign in to view

MichelleArk added 3 commits October 25, 2023 15:04

fix import

a89642a

move types_pb2.py from events to common/events

dbd02e5

move agate_helper into common

f44d704

MichelleArk mentioned this pull request Oct 26, 2023

[CT-3270] [Epic] Decouple dbt/adapters from core #8917

Closed

colin-rogers-dbt and others added 5 commits October 26, 2023 15:59

Add utils module (#8910)

1a82548

* moving types_pb2.py to common/events * split out utils into core/common/adapters * add changie

remove usage of dbt.config.PartialProject from dbt/adapters (#8909)

b9a603e

* remove usage of dbt.config.PartialProject from dbt/adapters * add changie --------- Co-authored-by: Colin <[email protected]>

Merge branch 'move-agate-client-to-common' into feature/decouple-adap…

200bcdc

…ters-from-core

move agate_helper unit tests under tests/unit/common

7de8930

move agate_helper into common (#8911)

af91666

* move agate_helper into common * add changie --------- Co-authored-by: Colin <[email protected]>

MichelleArk mentioned this pull request Nov 1, 2023

Decouple adapters from core #8666

Closed

4 tasks

MichelleArk and others added 4 commits November 1, 2023 12:00

Merge branch 'main' into feature/decouple-adapters-from-core

333120b

remove dbt.flags.MP_CONTEXT usage in dbt/adapters (#8931)

1260782

remove dbt.flags.LOG_CACHE_EVENTS usage in dbt/adapters (#8933)

c4ff280

Merge branch 'main' into feature/decouple-adapters-from-core

4ee9504

This was referenced Nov 7, 2023

[CT-3332] Remove usage of dbt.semver in dbt/adapters #9022

Closed

[CT-3334] Remove usage of dbt.contracts.graph.nodes.ConstraintType, ColumnLevelConstraint, ModelLevelConstraint in dbt/adapters #9024

Closed

colin-rogers-dbt and others added 10 commits November 10, 2023 09:54

Refactor Base Exceptions (#8989)

51b94b2

* moving types_pb2.py to common/events * Refactor Base Exceptions * update make_log_dir_if_missing to handle str * move remaining adapters exception imports to common/adapters --------- Co-authored-by: Michelle Ark <[email protected]>

Merge branch 'main' into feature/decouple-adapters-from-core

1d0a3e9

Remove usage of dbt.deprecations in dbt/adapters, enable core & adapt…

e56a5da

…er-specific (#9051)

Decouple adapter constraints from core (#9054)

2ab0f7b

* Move constraints to dbt.common * Move constraints to contracts folder, per review * Add a changelog entry.

move include/global_project to adapters (#8930)

615ad1f

remove adapter.get_compiler (#9134)

220f56d

Move adapter logger to adapters (#9165)

8bad75c

* moving types_pb2.py to common/events * Move AdapterLogger to adapter folder * add changie

Merge branch 'main' into feature/decouple-adapters-from-core

12bd1e8

delete accidentally merged types_pb2.py

294ad82

Move the semver package to common and alter references. (#9166)

373125e

* Move the semver package to common and alter references. * Alter leftover references to dbt.semver, this time using from syntax. --------- Co-authored-by: Mila Page <[email protected]>

Add config field to RelationConfig (#9300)

c08af13

* moving types_pb2.py to common/events * add config field to RelationConfig

colin-rogers-dbt reviewed Dec 19, 2023

View reviewed changes

core/dbt/common/exceptions/base.py

@@ -0,0 +1,275 @@

import builtins

Copy link

Contributor

colin-rogers-dbt Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look at split of exceptions into separate files

colin-rogers-dbt reviewed Dec 19, 2023

View reviewed changes

core/dbt/common/invocation.py

@@ -0,0 +1,12 @@

import uuid

_INVOCATION_ID = str(uuid.uuid4())

Copy link

Contributor

colin-rogers-dbt Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new invocation_id global

colin-rogers-dbt marked this pull request as ready for review December 19, 2023 17:49

colin-rogers-dbt requested review from a team as code owners December 19, 2023 17:49

colin-rogers-dbt requested review from heysweet and gshank and removed request for a team December 19, 2023 17:49

colin-rogers-dbt reviewed Dec 19, 2023

View reviewed changes

peterallenwebb approved these changes Dec 21, 2023

View reviewed changes

ChenyuLInx reviewed Dec 21, 2023

View reviewed changes

ChenyuLInx approved these changes Dec 22, 2023

View reviewed changes

colin-rogers-dbt and others added 2 commits December 22, 2023 09:03

Merge branch 'main' into feature/decouple-adapters-from-core

ac3b17f

colin-rogers-dbt merged commit 7763212 into main Dec 22, 2023
50 of 51 checks passed

HRusby mentioned this pull request Dec 26, 2023

[ADAP-1080] [Bug] Break in Dev build due to https://github.com/dbt-labs/dbt-core/pull/8906/files dbt-labs/dbt-snowflake#874

Closed

2 tasks

Topherhindman mentioned this pull request Jan 4, 2024

update link to materializations source code dbt-labs/docs.getdbt.com#4707

Merged

7 tasks

tlento mentioned this pull request Feb 26, 2024

Update dbt-semantic-interfaces dependency to compatible range #9671

Merged

dbeatty10 mentioned this pull request Jul 8, 2024

[Bug] Type object 'MacroFuzzEnvironment' has no attribute 'concat' #9896

Closed

2 tasks

This was referenced Sep 16, 2024

[CT-2869] Add new snapshot records when source removes a field #8207

Open

[Code docs] Several docs still reference parts of include/ that have been removed (to dbt-adapters) #10725

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/decouple adapters from core #8906

Feature/decouple adapters from core #8906

MichelleArk commented Oct 25, 2023 •

edited

Loading

This comment was marked as outdated.

colin-rogers-dbt Dec 19, 2023

colin-rogers-dbt Dec 19, 2023

colin-rogers-dbt Dec 19, 2023

colin-rogers-dbt Dec 19, 2023

colin-rogers-dbt Dec 19, 2023

colin-rogers-dbt Dec 19, 2023

ChenyuLInx left a comment

ChenyuLInx Dec 21, 2023

colin-rogers-dbt Dec 21, 2023

colin-rogers-dbt Dec 21, 2023

ChenyuLInx Dec 21, 2023

colin-rogers-dbt Dec 21, 2023

colin-rogers-dbt commented Dec 21, 2023

		@@ -93,11 +89,26 @@ class AdapterProtocol( # type: ignore[misc]
		def __init__(self, config: AdapterRequiredConfig) -> None:
		...

		@@ -0,0 +1,12 @@
		import uuid

		_INVOCATION_ID = str(uuid.uuid4())

		from dbt.common.clients.jinja import MacroProtocol


		class MacroResolverProtocol(Protocol):

		_EVENT_MANAGER: IEventManager = EventManager()


		def get_event_manager() -> IEventManager:

		@@ -0,0 +1,7 @@
		kind: Breaking Changes
		body: Rm --dry-run flag from 'dbt deps --add-package', in favor of just 'dbt deps

Feature/decouple adapters from core #8906

Feature/decouple adapters from core #8906

Conversation

MichelleArk commented Oct 25, 2023 • edited Loading

Problem

Solution

Checklist

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChenyuLInx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

colin-rogers-dbt commented Dec 21, 2023

MichelleArk commented Oct 25, 2023 •

edited

Loading