Skip to content
This repository has been archived by the owner on Jul 24, 2024. It is now read-only.

Abstract execution environment #44

Open
dinigo opened this issue Oct 14, 2021 · 1 comment
Open

Abstract execution environment #44

dinigo opened this issue Oct 14, 2021 · 1 comment

Comments

@dinigo
Copy link

dinigo commented Oct 14, 2021

As many out there might know every project is a whole different world. For this reason we find ourselves with a plethora of ways for running DBT. Though airflow-dbt is an elegant solution we sometimes want to run this on a k8s pod, on cloud build or some other task execution environment. This is motivated by decoupling, security, different dbt versions (yes, dbt SEMVER is not consistent and minor versions break backwards compatibility so this is a thing), or simply centralising different orchestration tasks in a task-runner because it is already configured.

To accomplish this it would be interesting to:

  1. Abstract the command generation from the execution:
  2. Allow for dependency injection, passing the hook we want to use and run accordingly, pass a python function that receives the command (ala GCSFileTransformOperator) ... open for discussion
@dinigo
Copy link
Author

dinigo commented Oct 21, 2021

Class structure goes like this:

image

You can tell whichever Operator you choose to use which DbtHook do you want to use. Using the interface DbtBaseHook ensures that they implement required methods

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
1 participant