Bugfix/redshift constant expressions (#91)

* temporarily materialize problem models as table * current fixes * add the using_abandoned_checkout variable * versioning, docs * add metafield enable disable config * changelog * updates * update changelog * rm macros since not used * updates * Update CHANGELOG.md Co-authored-by: Joe Markiewicz <[email protected]> * docs, add metafield config, update readme, changelog * update run script * new schema * udpate readme * format updates * Update CHANGELOG.md Co-authored-by: fivetran-catfritz <[email protected]> * update --------- Co-authored-by: Joe Markiewicz <[email protected]> Co-authored-by: fivetran-catfritz <[email protected]>
fivetran · Dec 2, 2024 · de1f561 · de1f561
1 parent f014e4b
commit de1f561
Show file tree

Hide file tree

Showing 17 changed files with 54 additions and 29 deletions.
diff --git a/.buildkite/scripts/run_models.sh b/.buildkite/scripts/run_models.sh
@@ -21,15 +21,15 @@ if [ "$db" = "databricks-sql" ]; then
 dbt seed --vars '{shopify_schema: shopify_source_integrations_tests_sqlw}' --target "$db" --full-refresh
 dbt run --vars '{shopify_schema: shopify_source_integrations_tests_sqlw}' --target "$db" --full-refresh
 dbt test --vars '{shopify_schema: shopify_source_integrations_tests_sqlw}' --target "$db"
-dbt run --vars '{shopify_schema: shopify_source_integrations_tests_sqlw, shopify_timezone: "America/New_York", shopify_using_fulfillment_event: true}' --target "$db" --full-refresh
+dbt run --vars '{shopify_schema: shopify_source_integrations_tests_sqlw, shopify_timezone: "America/New_York", shopify_using_fulfillment_event: true, shopify_using_abandoned_checkout: false, shopify_using_metafield: false}' --target "$db" --full-refresh
 dbt test --vars '{shopify_schema: shopify_source_integrations_tests_sqlw}' --target "$db"
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"
 
 else
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --vars '{shopify_timezone: "America/New_York", shopify_using_fulfillment_event: true}' --target "$db" --full-refresh
+dbt run --vars '{shopify_timezone: "America/New_York", shopify_using_fulfillment_event: true, shopify_using_abandoned_checkout: false, shopify_using_metafield: false}' --target "$db" --full-refresh
 dbt test --target "$db"
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"
 fi
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,17 @@
+# dbt_shopify_source v0.13.0
+
+[PR #91](https://github.com/fivetran/dbt_shopify_source/pull/91) includes the following changes:
+## Breaking Changes
+- Adds enable/disable config for the `metadata` staging model using the `shopify_using_metafield` variable (default `true`).
+- Adds enable/disable config for the `abandoned_checkout` staging models using the `shopify_using_abandoned_checkout` variable (default `true`):
+   - `stg_shopify__abandoned_checkout`
+   - `stg_shopify__abandoned_checkout_discount_code`
+   - `stg_shopify__abandoned_checkout_shipping_line`
+- For more information on how to enable/disable these tables, refer to the [README](https://github.com/fivetran/dbt_shopify_source/blob/main/README.md#step-4-disable-models-for-non-existent-sources). This will be a breaking change if you choose to disable these tables.
+
+## Under the Hood
+- Updates the `index` calculation in `stg_shopify__abandoned_checkout_discount_code` by removing the conditional logic for null scenarios now that a disable config has been added to the model.
+
 # dbt_shopify_source v0.12.1
 
 ## 🪲 Bug Fixes 🪛

diff --git a/README.md b/README.md
@@ -1,4 +1,6 @@
-<p align="center">
+# Shopify Source dbt Package ([Docs](https://fivetran.github.io/dbt_shopify_source/))
+
+<p align="left">
     <a alt="License"
         href="https://github.com/fivetran/dbt_shopify_source/blob/main/LICENSE">
         <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
@@ -13,7 +15,6 @@
         <img src="https://img.shields.io/badge/Fivetran_Quickstart_Compatible%3F-yes-green.svg" /></a>
 </p>
 
-# Shopify Source dbt Package ([Docs](https://fivetran.github.io/dbt_shopify_source/))
 ## What does this dbt package do?
 <!--section="shopify_source_model"-->
 - Materializes [Shopify staging tables](https://fivetran.github.io/dbt_shopify_source/#!/overview/github_source/models/?g_v=1) which leverage data in the format described by [this ERD](https://fivetran.com/docs/applications/shopify/#schemainformation). These staging tables clean, test, and prepare your Shopify data from [Fivetran's connector](https://fivetran.com/docs/applications/shopify) for analysis by doing the following:
@@ -44,7 +45,7 @@ If you  are **not** using the [Shopify transformation package](https://github.co
 ```yml
 packages:
   - package: fivetran/shopify_source
-    version: [">=0.12.0", "<0.13.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.13.0", "<0.14.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 ### Step 3: Define database and schema variables
@@ -72,18 +73,20 @@ vars:
 
 To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.
 
-### Step 4: Enable `fulfillment_event` data
+### Step 4: Disable models for non-existent sources
 
-The package takes into consideration that not every Shopify connector may have `fulfillment_event` data enabled. However, this table does hold valuable information that is leveraged in the `shopify__daily_shop` model in the transformation package. `fulfillment_event` data is **disabled by default**.
+The package takes into consideration that not every Shopify connector may have the `fulfillment_event`, `metadata`, or `abandoned_checkout` tables (including `abandoned_checkout`, `abandoned_checkout_discount_code`, and `abandoned_checkout_shipping_line`) and allows you to enable or disable the corresponding functionality. To enable/disable the modeling of the mentioned source tables and their downstream references, add the following variable to your `dbt_project.yml` file:
 
-Add the following variable to your `dbt_project.yml` file to enable the modeling of fulfillment events:
 ```yml
 # dbt_project.yml
 
 vars:
-    shopify_using_fulfillment_event: true # false by default
+    shopify_using_fulfillment_event: true # false by default. 
+    shopify_using_metafield: false  #true by default
+    shopify_using_abandoned_checkout: false # true by default. Setting to false will disable `abandoned_checkout`, `abandoned_checkout_discount_code`, and `abandoned_checkout_shipping_line`.
 ```
 
+
 ### Step 5: Setting your timezone
 By default, the data in your Shopify schema is in UTC. However, you may want reporting to reflect a specific timezone for more realistic analysis or data validation.
 

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'shopify_source'
-version: '0.12.1'
+version: '0.13.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/ci/sample.profiles.yml b/integration_tests/ci/sample.profiles.yml
@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: shopify_source_integration_tests_10
+      schema: shopify_source_integration_tests_12
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: shopify_source_integration_tests_10
+      schema: shopify_source_integration_tests_12
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: shopify_source_integration_tests_10
+      schema: shopify_source_integration_tests_12
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: shopify_source_integration_tests_10
+      schema: shopify_source_integration_tests_12
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: shopify_source_integration_tests_10
+      schema: shopify_source_integration_tests_12
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,10 +1,10 @@
 name: 'shopify_source_integration_tests'
-version: '0.12.1'
+version: '0.13.0'
 profile: 'integration_tests'
 config-version: 2
 
 vars:
-  shopify_schema: "shopify_source_integration_tests_10"
+  shopify_schema: shopify_source_integration_tests_12
   # shopify_using_fulfillment_event: true # enable when regenerating docs
   shopify_source:
     shopify_customer_identifier: "shopify_customer_data"

diff --git a/models/stg_shopify__abandoned_checkout.sql b/models/stg_shopify__abandoned_checkout.sql
@@ -1,3 +1,4 @@
+{{ config(enabled=var('shopify_using_abandoned_checkout', True)) }}
 
 with base as (
 

diff --git a/models/stg_shopify__abandoned_checkout_discount_code.sql b/models/stg_shopify__abandoned_checkout_discount_code.sql
@@ -1,3 +1,5 @@
+{{ config(enabled=var('shopify_using_abandoned_checkout', True)) }}
+
 with base as (
 
     select * 
@@ -20,6 +22,7 @@ fields as (
         }}
 
     from base
+
 ),
 
 final as (
@@ -34,13 +37,11 @@ final as (
         {{ dbt_date.convert_timezone(column='cast(updated_at as ' ~ dbt.type_timestamp() ~ ')', target_tz=var('shopify_timezone', "UTC"), source_tz="UTC") }} as updated_at,
         {{ dbt_date.convert_timezone(column='cast(_fivetran_synced as ' ~ dbt.type_timestamp() ~ ')', target_tz=var('shopify_timezone', "UTC"), source_tz="UTC") }} as _fivetran_synced,
         source_relation, 
-        case when checkout_id is null and code is null and index is null
-            then row_number() over(partition by source_relation order by source_relation)
-            else row_number() over(partition by checkout_id, upper(code), source_relation order by index desc)
-        end as index
+        row_number() over(partition by checkout_id, upper(code), source_relation order by index desc) as index
+
 
     from fields
-    
+
 )
 
 select *

diff --git a/models/stg_shopify__abandoned_checkout_shipping_line.sql b/models/stg_shopify__abandoned_checkout_shipping_line.sql
@@ -1,3 +1,4 @@
+{{ config(enabled=var('shopify_using_abandoned_checkout', True)) }}
 
 with base as (
 

diff --git a/models/stg_shopify__metafield.sql b/models/stg_shopify__metafield.sql
@@ -1,3 +1,4 @@
+{{ config(enabled=var('shopify_using_metafield', True)) }}
 
 with base as (
 
@@ -38,10 +39,7 @@ final as (
         {{ dbt_date.convert_timezone(column='cast(updated_at as ' ~ dbt.type_timestamp() ~ ')', target_tz=var('shopify_timezone', "UTC"), source_tz="UTC") }} as updated_at,
         {{ dbt_date.convert_timezone(column='cast(_fivetran_synced as ' ~ dbt.type_timestamp() ~ ')', target_tz=var('shopify_timezone', "UTC"), source_tz="UTC") }} as _fivetran_synced,
         lower({{ dbt.concat(["namespace","'_'","key"]) }}) as metafield_reference,
-        case when id is null and updated_at is null
-            then row_number() over(partition by source_relation order by source_relation) = 1
-            else row_number() over(partition by id, source_relation order by updated_at desc) = 1
-        end as is_most_recent_record,
+        row_number() over(partition by id, source_relation order by updated_at desc) = 1 as is_most_recent_record,
         source_relation
 
     from fields

diff --git a/models/tmp/stg_shopify__abandoned_checkout_discount_code_tmp.sql b/models/tmp/stg_shopify__abandoned_checkout_discount_code_tmp.sql
@@ -1,3 +1,5 @@
+{{ config(enabled=var('shopify_using_abandoned_checkout', True)) }}
+
 {{
     fivetran_utils.union_data(
         table_identifier='abandoned_checkout_discount_code', 

diff --git a/models/tmp/stg_shopify__abandoned_checkout_shipping_line_tmp.sql b/models/tmp/stg_shopify__abandoned_checkout_shipping_line_tmp.sql
@@ -1,3 +1,5 @@
+{{ config(enabled=var('shopify_using_abandoned_checkout', True)) }}
+
 {{
     fivetran_utils.union_data(
         table_identifier='abandoned_checkout_shipping_line', 

diff --git a/models/tmp/stg_shopify__abandoned_checkout_tmp.sql b/models/tmp/stg_shopify__abandoned_checkout_tmp.sql
@@ -1,3 +1,5 @@
+{{ config(enabled=var('shopify_using_abandoned_checkout', True)) }}
+
 {{
     fivetran_utils.union_data(
         table_identifier='abandoned_checkout', 

diff --git a/models/tmp/stg_shopify__metafield_tmp.sql b/models/tmp/stg_shopify__metafield_tmp.sql
@@ -1,3 +1,5 @@
+{{ config(enabled=var('shopify_using_metafield', True)) }}
+
 {{
     fivetran_utils.union_data(
         table_identifier='metafield',
Original file line number	Diff line number	Diff line change
		@@ -1,3 +1,4 @@
		{{ config(enabled=var('shopify_using_abandoned_checkout', True)) }}

		with base as (

Expand Down