bugfix/is-incremental-compatible (#4)

* bugfix/is-incremental-compatible * PR link ref in CHANGELOG * Update README.md Co-authored-by: fivetran-catfritz <[email protected]> * Update integration_tests/tests/consistency/consistency_events_enhanced.sql Co-authored-by: fivetran-catfritz <[email protected]> * review updates * Apply suggestions from code review Co-authored-by: Avinash Kunnath <[email protected]> * address release review notes --------- Co-authored-by: fivetran-catfritz <[email protected]> Co-authored-by: Avinash Kunnath <[email protected]>
fivetran · Jul 17, 2024 · 8cf68f5 · 8cf68f5
1 parent 7972896
commit 8cf68f5
Show file tree

Hide file tree

Showing 16 changed files with 190 additions and 55 deletions.
diff --git a/.buildkite/hooks/pre-command b/.buildkite/hooks/pre-command
@@ -22,4 +22,6 @@ export CI_SNOWFLAKE_DBT_WAREHOUSE=$(gcloud secrets versions access latest --secr
 export CI_DATABRICKS_DBT_HOST=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_HOST" --project="dbt-package-testing-363917")
 export CI_DATABRICKS_DBT_HTTP_PATH=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_HTTP_PATH" --project="dbt-package-testing-363917")
 export CI_DATABRICKS_DBT_TOKEN=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_TOKEN" --project="dbt-package-testing-363917")
-export CI_DATABRICKS_DBT_CATALOG=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_CATALOG" --project="dbt-package-testing-363917")
+export CI_DATABRICKS_DBT_CATALOG=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_CATALOG" --project="dbt-package-testing-363917")
+export CI_DATABRICKS_SQL_DBT_HTTP_PATH=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_SQL_DBT_HTTP_PATH" --project="dbt-package-testing-363917")
+export CI_DATABRICKS_SQL_DBT_TOKEN=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_SQL_DBT_TOKEN" --project="dbt-package-testing-363917")
diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml
@@ -71,4 +71,19 @@ steps:
             - "CI_DATABRICKS_DBT_TOKEN"
             - "CI_DATABRICKS_DBT_CATALOG"
     commands: |
-      bash .buildkite/scripts/run_models.sh databricks
+      bash .buildkite/scripts/run_models.sh databricks
+
+  - label: ":databricks: :database: Run Tests - Databricks SQL Warehouse"
+    key: "run_dbt_databricks_sql"
+    plugins:
+      - docker#v3.13.0:
+          image: "python:3.8"
+          shell: [ "/bin/bash", "-e", "-c" ]
+          environment:
+            - "BASH_ENV=/tmp/.bashrc"
+            - "CI_DATABRICKS_DBT_HOST"
+            - "CI_DATABRICKS_SQL_DBT_HTTP_PATH"
+            - "CI_DATABRICKS_SQL_DBT_TOKEN"
+            - "CI_DATABRICKS_DBT_CATALOG"
+    commands: |
+      bash .buildkite/scripts/run_models.sh databricks-sql
diff --git a/.buildkite/scripts/run_models.sh b/.buildkite/scripts/run_models.sh
@@ -16,12 +16,23 @@ db=$1
 echo `pwd`
 cd integration_tests
 dbt deps
+if [ "$db" = "databricks-sql" ]; then
+dbt seed --vars '{salesforce_marketing_cloud_schema: sfmc_sqlw_tests}' --target "$db" --full-refresh
+dbt run --vars '{salesforce_marketing_cloud_schema: sfmc_sqlw_tests}' --target "$db" --full-refresh
+dbt test --vars '{salesforce_marketing_cloud_schema: sfmc_sqlw_tests}' --target "$db"
+dbt run --vars '{salesforce_marketing_cloud_schema: sfmc_sqlw_tests}' --target "$db"
+dbt test --vars '{salesforce_marketing_cloud_schema: sfmc_sqlw_tests}' --target "$db"
+dbt run --vars '{salesforce_marketing_cloud_schema: sfmc_sqlw_tests, salesforce_marketing_cloud__link_enabled: false, salesforce_marketing_cloud__list_enabled: false}' --full-refresh --target "$db"
+dbt run --vars '{salesforce_marketing_cloud_schema: sfmc_sqlw_tests, salesforce_marketing_cloud__link_enabled: false, salesforce_marketing_cloud__list_enabled: false}' --target "$db"
+dbt test --vars '{salesforce_marketing_cloud_schema: sfmc_sqlw_tests}' --target "$db"
+else
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
 dbt run --target "$db"
 dbt test --target "$db"
 dbt run --vars '{salesforce_marketing_cloud__link_enabled: false, salesforce_marketing_cloud__list_enabled: false}' --full-refresh --target "$db"
+dbt run --vars '{salesforce_marketing_cloud__link_enabled: false, salesforce_marketing_cloud__list_enabled: false}' --target "$db"
 dbt test --target "$db"
-
+fi
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,21 @@
+# dbt_salesforce_marketing_cloud v0.2.0
+
+[PR #4](https://github.com/fivetran/dbt_salesforce_marketing_cloud/pull/4) includes the following updates. Please be aware these changes only impact Databricks warehouse users:
+
+## 🚨 Breaking Changes 🚨
+> ⚠️ Since the following changes result in the table format changing for Databricks users, we recommend running a `--full-refresh` after upgrading to this version to avoid possible incremental failures.
+- For Databricks All-Purpose clusters, the `salesforce_marketing_cloud__events_enhanced` model will now be materialized using the delta table format (previously parquet). 
+  - Delta tables are generally more performant than parquet and are also more widely available for Databricks users. Previously, the parquet file format was causing compilation issues on customers' managed tables.
+
+## Documentation
+- Added details to the README to highlight the incremental strategies used within the `salesforce_marketing_cloud__events_enhanced` model.
+
+## Under the Hood
+- The `is_incremental_compatible` macro has been added to the package. This macro will return `true` if the Databricks runtime being used is an all-purpose cluster **or** if any other non-Databricks supported destination is being used.
+  - This update was applied as there are other Databricks runtimes (ie. sql warehouse, endpoint, and external runtime) which do not support the `insert_overwrite` incremental strategy used in the `salesforce_marketing_cloud__events_enhanced` model. 
+- In addition to the above, for Databricks users the `salesforce_marketing_cloud__events_enhanced` model will now leverage the incremental strategy only if the Databricks runtime is all-purpose. Otherwise, all other Databricks runtimes will not leverage an incremental strategy.
+- Added validation tests to the `integration_tests` folder to ensure the consistency and integrity of the `salesforce_marketing_cloud__events_enhanced` model for subsequent updates.
+
 # dbt_salesforce_marketing_cloud v0.1.0
 🎉 This is the initial release of this package! 🎉
 

diff --git a/README.md b/README.md
@@ -56,14 +56,24 @@ dispatch:
     search_order: ['spark_utils', 'dbt_utils']
 ```
 
+### Database Incremental Strategies 
+The `salesforce_marketing_cloud__events_enhanced` model in this package is materialized incrementally and is configured to work with the different strategies available to each supported warehouse.
+
+For **BigQuery** and **Databricks All-Purpose Cluster runtime** destinations, we have chosen `insert_overwrite` as the default strategy, which benefits from the partitioning capability. 
+> For all other Databricks runtimes, models are materialized as tables without support for incremental runs.
+
+For **Snowflake**, **Redshift**, and **Postgres** databases, we have chosen `delete+insert` as the default strategy.  
+
+> Regardless of strategy, we recommend that users periodically run a `--full-refresh` to ensure a high level of data quality.
+
 ## Step 2: Install the package
 Include the following Salesforce Marketing Cloud package version in your `packages.yml` file:
 > [!TIP]
 > Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
 ```yml
 packages:
   - package: fivetran/salesforce_marketing_cloud
-    version: [">=0.1.0", "<0.2.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.2.0", "<0.3.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 ## Step 3: Define database and schema variables

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,6 +1,6 @@
 config-version: 2
 name: 'salesforce_marketing_cloud'
-version: '0.1.0'
+version: '0.2.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:

diff --git a/docs/catalog.json b/docs/catalog.json