-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[dagster-airlift] Operator Migration guides
- Loading branch information
Showing
5 changed files
with
82 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
53 changes: 53 additions & 0 deletions
53
docs/content/integrations/airlift/operator-migration/bash-operator-dbt.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
# Operator migration guides: Migrating usage of `BashOperator` for `dbt` | ||
|
||
In this page, we'll explain migrating an Airflow `BashOperator` that runs a `dbt` command to Dagster. | ||
|
||
### Background | ||
|
||
In Airflow, you might have a `BashOperator` that runs a `dbt` command. For example, you might have a task that runs `dbt run` to build your dbt models. | ||
|
||
```python file=/integrations/airlift/operator_migration/bash_operator_dbt.py | ||
from airflow.operators.bash import BashOperator | ||
|
||
run_dbt_model = BashOperator(task_id="build_dbt_models", bash_command="dbt run") | ||
``` | ||
|
||
### Dagster equivalent | ||
|
||
The Dagster equivalent is to instead use the `dagster-dbt` library to run commands against your dbt project. Here would be the equivalent code in Dagster: | ||
|
||
```python file=/integrations/airlift/operator_migration/using_dbt_assets.py | ||
from dagster_dbt import DbtCliResource, DbtProject, dbt_assets | ||
|
||
from dagster import AssetExecutionContext | ||
|
||
project = DbtProject(project_dir="path/to/dbt_project") | ||
|
||
|
||
@dbt_assets(manifest=project.manifest_path, project=project) | ||
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): | ||
yield from dbt.cli(["run"], context=context).stream() | ||
``` | ||
|
||
### Migrating the operator | ||
|
||
Migrating the operator breaks down into a few steps: | ||
|
||
1. Making the dbt project available to both your Airflow and Dagster deployments. | ||
2. Writing a `@dbt_asset`-decorated function which runs your dbt commands. | ||
3. Using `dagster-airlift` to proxy execution of the original task to Dagster. | ||
|
||
### Step 1: Making the dbt project available & building manifest | ||
|
||
First, you'll need to make the dbt project available to the Dagster runtime and build the manifest. | ||
|
||
- If you're building your Dagster deployment in a monorepo alongside your dbt and Airflow projects, you can follow this guide: [Monorepo setup](/integrations/dbt/reference#deploying-a-dagster-project-with-a-dbt-project). | ||
- If you're deploying within a separate repository, you can follow this guide: [Separate repository setup](/integrations/dbt/reference#deploying-a-dbt-project-from-a-separate-git-repository). | ||
|
||
### Step 2: Writing a `@dbt_asset`-decorated function | ||
|
||
Once your dbt project is available, you can write a function that runs your dbt commands using the <PyObject object="dbt_assets" module="dagster_dbt"/> decorator and <PyObject object="DbtCliResource" module="dagster_dbt"/>. Most dbt CLI commands and flags are supported - to see a full guide on using `dbt_assets`, check out the [dagster-dbt guide](/integrations/dbt/quickstart). | ||
|
||
### Step 3: Using `dagster-airlift` to proxy execution | ||
|
||
Finally, you can use `dagster-airlift` to proxy the execution of the original task to Dagster. The [dagster-airlift migration guide](/integrations/airlift/tutorial/overview) details this process. |
12 changes: 12 additions & 0 deletions
12
docs/content/integrations/airlift/operator-migration/overview.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
# Operator Migration Reference | ||
|
||
This page contains a collection of reference materials for migrating usage of common Airflow operator types to Dagster. | ||
|
||
## References | ||
|
||
<ArticleList> | ||
<ArticleListItem | ||
title="Bash Operator for dbt" | ||
href="/integrations/airlift/operator-migration/bash-operator-dbt" | ||
></ArticleListItem> | ||
</ArticleList> |
4 changes: 1 addition & 3 deletions
4
.../docs_snippets/docs_snippets/integrations/airlift/operator_migration/bash_operator_dbt.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,3 @@ | ||
from airflow.operators.bash import BashOperator | ||
|
||
run_dbt_model = BashOperator( | ||
task_id="build_dbt_models", bash_command="dbt run", dag=... | ||
) | ||
run_dbt_model = BashOperator(task_id="build_dbt_models", bash_command="dbt run") |