Merge pull request #70 from fivetran/feature/shopify-api-updates

Feature/shopify api updates

CHANGELOG.md

@@ -1,3 +1,22 @@
+# dbt_shopify_source v0.9.0
+## Breaking Changes
+- In [June 2023](https://fivetran.com/docs/applications/shopify/changelog#june2023) the Shopify connector received an update which upgraded the connector to be compatible with the new [2023-04 Shopify API](https://shopify.dev/docs/api). As a result, the following fields have been removed as they were deprecated in the API upgrade: ([PR #70](https://github.com/fivetran/dbt_shopify_source/pull/70))
+
+| **model** | **field removed** |
+|-------|--------------|
+| [stg_shopify__customer](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__customer) | `lifetime_duration` |
+| [stg_shopify__order_line](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__order_line) | `fulfillment_service` |
+| [stg_shopify__order_line](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__order_line) | `destination_location_*` fields |
+| [stg_shopify__order_line](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__order_line) | `origin_location_*` fields |
+| [stg_shopify__order](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__order) | `total_price_usd` |
+| [stg_shopify__order](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__order) | `processing_method` |
+
+## Under the Hood
+- Removed `databricks` from the shopify_database configuration in the `src_shopify.yml` to allow Databricks Unity catalog users to define a unity Catalog as a database. ([PR #70](https://github.com/fivetran/dbt_shopify_source/pull/70))
+
+## Documentation Updates
+- Documentation provided in the README for how to connect sources when leveraging the union schema/database feature. ([PR #70](https://github.com/fivetran/dbt_shopify_source/pull/70))
+
 # dbt_shopify_source v0.8.3
 
 ## Bug Fixes 🐛 🪛 

README.md

@@ -44,7 +44,7 @@ If you  are **not** using the [Shopify transformation package](https://github.co
 ```yml
 packages:
   - package: fivetran/shopify_source
-    version: [">=0.8.0", "<0.9.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.9.0", "<0.10.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 ## Step 3: Define database and schema variables
@@ -68,6 +68,10 @@ vars:
     shopify_union_databases: ['shopify_usa','shopify_canada'] # use this if the data is in different databases/projects but uses the same schema name
 ```
 
+Please be aware that the native `source.yml` connection set up in the package will not function when the union schema/database feature is utilized. Although the data will be correctly combined, you will not observe the sources linked to the package models in the Directed Acyclic Graph (DAG). This happens because the package includes only one defined `source.yml`.
+
+To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.
+
 ## Step 4: Enable `fulfillment_event` data
 
 The package takes into consideration that not every Shopify connector may have `fulfillment_event` data enabled. However, this table does hold valuable information that is leveraged in the `shopify__daily_shop` model in the transformation package. `fulfillment_event` data is **disabled by default**. 
@@ -148,7 +152,7 @@ If you are making use of the `shopify_union_schemas` or `shopify_union_databases
 
 ### Disable Compiler Warnings for Empty Tables
 
-Empty staging models are created in the Shopify schema dyanmically if the respective source tables do not exist in your raw source schema. For example, if your shop has not incurred any refunds, you will not have a `refund` table yet until you do refund an order, and the package will create an empty `stg_shopify__refund` model.
+Empty staging models are created in the Shopify schema dynamically if the respective source tables do not exist in your raw source schema. For example, if your shop has not incurred any refunds, you will not have a `refund` table yet until you do refund an order, and the package will create an empty `stg_shopify__refund` model.
 
 The source package will will return **completely** empty staging models (ie `limit 0`) if these source tables do not exist in your Shopify schema yet, and the transform package will work seamlessly with these empty models. Once an anticipated source table exists in your schema, the source and transform packages will automatically reference the new populated table(s). ([example](https://github.com/fivetran/dbt_shopify_source/blob/main/models/tmp/stg_shopify__refund_tmp.sql)). 
 

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'shopify_source'
-version: '0.8.3'
+version: '0.9.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'shopify_source_integration_tests'
-version: '0.8.3'
+version: '0.9.0'
 profile: 'integration_tests'
 config-version: 2
 

macros/get_customer_columns.sql

@@ -22,7 +22,6 @@
     {"name": "note", "datatype": dbt.type_string()},
     {"name": "accepts_marketing_updated_at", "datatype": dbt.type_timestamp()},
     {"name": "marketing_opt_in_level", "datatype": dbt.type_string()},
-    {"name": "lifetime_duration", "datatype": dbt.type_string()},
     {"name": "currency", "datatype": dbt.type_string()}
 ] %}
 

macros/get_order_columns.sql

@@ -11,7 +11,6 @@
     {"name": "total_line_items_price_set", "datatype": dbt.type_string()},
     {"name": "total_price", "datatype": dbt.type_float()},
     {"name": "total_price_set", "datatype": dbt.type_string()},
-    {"name": "total_price_usd", "datatype": dbt.type_float()},
     {"name": "total_tax_set", "datatype": dbt.type_string()},
     {"name": "total_tax", "datatype": dbt.type_float()},
     {"name": "source_name", "datatype": dbt.type_string()},
@@ -36,7 +35,6 @@
     {"name": "email", "datatype": dbt.type_string()},
     {"name": "financial_status", "datatype": dbt.type_string()},
     {"name": "fulfillment_status", "datatype": dbt.type_string()},
-    {"name": "processing_method", "datatype": dbt.type_string()},
     {"name": "referring_site", "datatype": dbt.type_string()},
     {"name": "billing_address_address_1", "datatype": dbt.type_string()},
     {"name": "billing_address_address_2", "datatype": dbt.type_string()},

macros/get_order_line_columns.sql

@@ -3,7 +3,6 @@
 {% set columns = [
     {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
     {"name": "fulfillable_quantity", "datatype": dbt.type_numeric()},
-    {"name": "fulfillment_service", "datatype": dbt.type_string()},
     {"name": "fulfillment_status", "datatype": dbt.type_string()},
     {"name": "gift_card", "datatype": "boolean"},
     {"name": "grams", "datatype": dbt.type_numeric()},
@@ -28,23 +27,7 @@
     {"name": "variant_title", "datatype": dbt.type_string()},
     {"name": "variant_inventory_management", "datatype": dbt.type_string()},
     {"name": "vendor", "datatype": dbt.type_string()},
-    {"name": "properties", "datatype": dbt.type_string()},
-    {"name": "destination_location_address_1", "datatype": dbt.type_string()},
-    {"name": "destination_location_address_2", "datatype": dbt.type_string()},
-    {"name": "destination_location_city", "datatype": dbt.type_string()},
-    {"name": "destination_location_country_code", "datatype": dbt.type_string()},
-    {"name": "destination_location_id", "datatype": dbt.type_int()},
-    {"name": "destination_location_name", "datatype": dbt.type_string()},
-    {"name": "destination_location_province_code", "datatype": dbt.type_string()},
-    {"name": "destination_location_zip", "datatype": dbt.type_string()},
-    {"name": "origin_location_address_1", "datatype": dbt.type_string()},
-    {"name": "origin_location_address_2", "datatype": dbt.type_string()},
-    {"name": "origin_location_city", "datatype": dbt.type_string()},
-    {"name": "origin_location_country_code", "datatype": dbt.type_string()},
-    {"name": "origin_location_id", "datatype": dbt.type_int()},
-    {"name": "origin_location_name", "datatype": dbt.type_string()},
-    {"name": "origin_location_province_code", "datatype": dbt.type_string()},
-    {"name": "origin_location_zip", "datatype": dbt.type_string()}
+    {"name": "properties", "datatype": dbt.type_string()}
 ] %}
 
 {{ fivetran_utils.add_pass_through_columns(columns, var('order_line_pass_through_columns')) }}

models/src_shopify.yml

@@ -3,7 +3,7 @@ version: 2
 sources:
   - name: shopify # This source will only be used if you are using a single Shopify source connector. If multiple sources are being unioned, their tables will be directly referenced via adatper.get_relation
     schema: "{{ var('shopify_schema', 'shopify') }}"
-    database: "{% if target.type not in ('spark', 'databricks') %}{{ var('shopify_database', target.database) }}{% endif %}"
+    database: "{% if target.type not in ('spark') %}{{ var('shopify_database', target.database) }}{% endif %}"
     tables:
       - name: order
         description: Each record represents an order in Shopify.
@@ -87,8 +87,6 @@ sources:
             description: The order 's position in the shop's count of orders starting at 1001. Order numbers are sequential and start at 1001.
           - name: processed_at
             description: The date and time when an order was processed. This value is the date that appears on your orders and that's used in the analytic reports.
-          - name: processing_method
-            description: How the payment was processed.
           - name: referring_site
             description: The website where the customer clicked a link to the shop.
           - name: shipping_address_address_1
@@ -177,12 +175,8 @@ sources:
             description: The total of all line item prices in shop and presentment currencies.
           - name: total_price_set
             description: The total price of the order in shop and presentment currencies.
-          - name: total_price_usd
-            description: The sum of all line item prices, discounts, shipping, taxes, and tips in USD. Must be positive.
           - name: is_confirmed
             description: Whether the order is confirmed.
-          - name: current_total_price
-            description: The current total price of the order in the shop currency. The value of this field reflects order edits, returns, and refunds.
           - name: source_identifier
             description: The ID of the order placed on the originating platform. This value doesn't correspond to the Shopify ID that's generated from a completed draft.
 
@@ -230,8 +224,6 @@ sources:
             description: Deprecated. The package will coalesce with `email_marketing_consent_consent_updated_at`. 
           - name: marketing_opt_in_level
             description: Deprecated. The package will coalesce with `email_marketing_consent_opt_in_level`.
-          - name: lifetime_duration
-            description: The amount of time since the customer was first added to the store.
           - name: _fivetran_deleted
             description: "{{ doc('_fivetran_deleted') }}"
           - name: note
@@ -247,8 +239,6 @@ sources:
             description: "{{ doc('_fivetran_synced') }}"
           - name: fulfillable_quantity
             description: "The amount available to fulfill, calculated as follows: quantity - max(refunded_quantity, fulfilled_quantity) - pending_fulfilled_quantity - open_fulfilled_quantity"
-          - name: fulfillment_service
-            description: The service provider that's fulfilling the item.
           - name: fulfillment_status
             description: How far along an order is in terms line items fulfilled.
           - name: gift_card
@@ -284,13 +274,13 @@ sources:
           - name: index
             description: Index of the order line.
           - name: pre_tax_price
-            description: The pre tax preice of the line item in shop currency.
+            description: The pre tax price of the line item in shop currency.
           - name: pre_tax_price_set
-            description: he pre tax preice of the line item in shop currency and presentment currency.
+            description: The pre tax price of the line item in shop currency and presentment currency.
           - name: price_set
             description: The price of the line item in shop and presentment currencies.
           - name: tax_code
-            descripton: Tax code applied to the line item. As multiple taxes can apply to a line item, we recommend referring to `stg_shopify__tax_line`.
+            description: Tax code applied to the line item. As multiple taxes can apply to a line item, we recommend referring to `stg_shopify__tax_line`.
           - name: total_discount_set
             description: The total amount allocated to the line item in the presentment currency.
           - name: variant_title
@@ -299,40 +289,8 @@ sources:
             description: The fulfillment service that tracks the number of items in stock for the product variant.
           - name: properties
             description: Line item properties.
-          - name: destination_location_address_1
-            description: The street address of the assigned location.
-          - name: destination_location_address_2
-            description: An optional additional field for the street address of the assigned location.
-          - name: destination_location_city
-            description: City of the destination where the items should be sent.
-          - name: destination_location_country_code
-            description: Two-letter country code (ISO 3166-1 format) of the destination where the items should be sent.
-          - name: destination_location_id
-            description: ID of the destination where the items should be sent.
-          - name: destination_location_name
-            description: Name of the destination where the items should be sent.
-          - name: destination_location_province_code
-            description: Two-letter province code of the destination where the items should be sent.
-          - name: destination_location_zip
-            description: Zip code of the destination where the items should be sent.
-          - name: origin_location_address_1
-            description: The street address of the item's supplier.
-          - name: origin_location_address_2
-            description: The suite number of the item's supplier.
-          - name: origin_location_city
-            description: The city of the item's supplier.
-          - name: origin_location_country_code
-            description: The two-letter code (ISO 3166-1 format) for the country of the item's supplier.
-          - name: origin_location_id
-            description: The location ID of the line item’s fulfillment origin. Used by Shopify to calculate applicable taxes. This is not the ID of the location where the order was placed. You can use the FulfillmentOrder resource to determine the location an item will be sourced from.
-          - name: origin_location_name
-            description: The name of the item's supplier.
-          - name: origin_location_province_code
-            description: The two-letter abbreviation for the region of the item's supplier.
-          - name: origin_location_zip
-            description: The zip of the item's supplier.
           - name: tax_code
-            descripton: Tax code applied to the line item. As multiple taxes can apply to a line item, we recommend referring to `stg_shopify__tax_line`.
+            description: Tax code applied to the line item. As multiple taxes can apply to a line item, we recommend referring to `stg_shopify__tax_line`.
 
       - name: order_line_refund
         identifier: "{{ var('shopify_order_line_refund_identifier', 'order_line_refund') }}"