v0.13.0 dbt_shopify_source

PR #91 includes the following changes:

Breaking Changes

• Adds enable/disable config for the metadata staging model using the shopify_using_metafield variable (default true).
• Adds enable/disable config for the abandoned_checkout staging models using the shopify_using_abandoned_checkout variable (default true):
  • stg_shopify__abandoned_checkout
  • stg_shopify__abandoned_checkout_discount_code
  • stg_shopify__abandoned_checkout_shipping_line
• For more information on how to enable/disable these tables, refer to the README. This will be a breaking change if you choose to disable these tables.

Under the Hood

• Updates the index calculation in stg_shopify__abandoned_checkout_discount_code by removing the conditional logic for null scenarios now that a disable config has been added to the model.

Full Changelog: v0.12.1...v0.13.0

v0.12.1 dbt_shopify_source

🪲 Bug Fixes 🪛

• Added support for a new delayed fulfillment event status from Shopify. delayed has been added to the accepted_values test on stg_shopify__fulfillment_event (PR #84).
• Added product_id to the unique combination_of_columns test for the stg_shopify__product_image model (PR #86).

Contributors

• @ryan-brainforge (PR #86)
• @shreveasaurus (PR #84)

Full Changelog: v0.12.0...v0.12.1

v0.12.0 dbt_shopify_source

PR #79 introduces the following changes:

🚨 Breaking Changes 🚨

• To reduce storage, updated default materialization of staging models from tables to views.
  • Note that stg_shopify__metafield will still be materialized as a table for downstream use.

⚠️ Running a --full-refresh will be required if you have previously run these staging models as tables and get the following error:

Trying to create view <model path> but it currently exists as a table. Either drop <model path> manually, or run dbt with `--full-refresh` and dbt will drop it for you.

Under the Hood

• Updated the maintainer PR template to the current format.
• Added integration testing pipeline for Databricks SQL Warehouse.

PR #81 introduces the following changes:

🪲 Bug Fixes 🪛

• Removed the index filter in stg_shopify__order_discount_code, as we were erroneously filtering out multiple discounts for an order since index is meant to pair with order_id as the unique identifier for this source.
• Added index as a field in stg_shopify__order_discount_code, as it is part of the primary key.

📝 Documentation Updates 📝

• Added index documentation to our src_shopify.yml and stg_shopify.yml.
• Updated the unique_combination_of_columns test on stg_shopify__order_discount_code to correctly check on index with order_id and source_relation rather than code.

🔧 Under the Hood 🔩

• Updated the pull request templates.

Full Changelog: v0.11.0...v0.12.0

v0.11.0 dbt_shopify_source

PR #78 introduces the following changes:

🚨 Breaking Changes 🚨

• Added source_relation to the partition_by clauses that determine the is_most_recent_record in the stg_shopify__metafield table.
• Added source_relation to the partition_by clauses that determines the index in the stg_shopify__abandoned_checkout_discount_code table.
• If the user is leveraging the union feature, this could change data values.

🐛 Bug Fixes 🪛

• Updated partition logic in stg_shopify__metafield and stg_shopify__abandoned_checkout_discount_code to account for null table Redshift errors when handling null field cases.

🚘 Under The Hood 🚘

• Included auto-releaser GitHub Actions workflow to automate future releases.
• Added additional casting in seed dependencies for above models integration_tests/dbt_project.yml to ensure local testing passed on null cases.

Full Changelog: v0.10.0...v0.11.0

v0.10.0 dbt_shopify_source

🚨 Breaking Changes 🚨

• This release will be a breaking change due to the removal of below dependencies.

Dependency Updates

• Removes the dependency on dbt-expectations and updates the dbt-date dependency to the latest version. (PR #75)

Full Changelog: v0.9.0...v0.10.0

v0.9.0 dbt_shopify_source

Breaking Changes

• In June 2023 the Shopify connector received an update which upgraded the connector to be compatible with the new 2023-04 Shopify API. As a result, the following fields have been removed as they were deprecated in the API upgrade: (PR #70)

model	field removed
stg_shopify__customer	lifetime_duration
stg_shopify__order_line	fulfillment_service
stg_shopify__order_line	destination_location_* fields
stg_shopify__order_line	origin_location_* fields
stg_shopify__order	total_price_usd
stg_shopify__order	processing_method

Under the Hood

• Removed databricks from the shopify_database configuration in the src_shopify.yml to allow Databricks Unity catalog users to define a unity Catalog as a database. (PR #70)

Documentation Updates

• Documentation provided in the README for how to connect sources when leveraging the union schema/database feature. (PR #70)

Full Changelog: v0.8.3...v0.9.0

v0.8.3 dbt_shopify_source

Happy World Refrigeration Day 🧊 ! Did you know Albert Einstein co-invented a refrigerator?

Hopefully you can track more successfully refrigerator orders to your company with the latest updates to the dbt_shopify_source package.

Bug Fixes 🐛 🪛

PR #69 includes the following fixes:

• Lower casing metafield_reference field in stg_shopify__metafield to fix metafield table breakages upstream when the key field has different casing for otherwise identical strings.
• Lower casing owner_resource field in stg_shopify__metafield to ensure identical value fields with different casing are then correctly pivoted together upstream in the shopify transformation package get_metafields macro.

Under the Hood:

• Incorporated the new fivetran_utils.drop_schemas_automation macro into the end of each Buildkite integration test job. (PR #65)
• Updated the pull request templates. (PR #65)

Contributors

• @ZCrookston & @FridayPush (Issue #64)

v0.8.2 dbt_shopify_source

Bug Fixes

PR #59 introduces the following changes:

• The fivetan_utils.union_data macro has been expanded to handle checking if a source table exists. Previously in the Shopify source package, this check happened outside of the macro and depended on the user having a defined shopify source. If the package anticipates a table that you do not have in any schema or database, it will return a completely empty table (ie limit 0) that will work seamlessly with downstream transformations.
  • A compilation message will be raised when a staging model is completely empty. This compiler warning can be turned off by the end user by setting the fivetran__remove_empty_table_warnings variable to True (see https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source for details).
• A uniqueness test has been placed on the order_line_id, index, and source_relation columns in stg_shopify__tax_line, as it was previously missing a uniqueness test.

Contributors

• @dfagnan (Issue #57)

v0.8.1...v0.8.2

v0.8.1 dbt_shopify_source

Happy Thursday!

Bug Fixes

• Addresses Issue #54, in which the deprecated discount_id field was used instead of code in stg_shopify__abandoned_checkout_discount__code (PR #56).

Full Changelog: v0.8.0...v0.8.1

v0.8.0 dbt_shopify_source

Lots of new features ahead!! We've revamped the package to keep up-to-date with new additions to the Shopify connector and feedback from the community.

This release includes 🚨 Breaking Changes 🚨.

Documentation

• Created the DECISIONLOG to log discussions and opinionated stances we took in designing the package (PR #45).
• README updated for easier package use and navigation (PR #38).

Under the Hood

• Ensured Postgres compatibility (PR #38).
• Got rid of the shopify__using_order_adjustment, shopify__using_order_line_refund, and shopify__using_refund variables. Instead, the package will automatically create empty versions of the related models until the source refund, order_line_refund, and order_adjustment tables exist in your schema. See DECISIONLOG for more details (PR #45).
• Adjusts the organization of the get_<table>_columns() macros (PR #39, PR #40).

Feature Updates

• Addition of the shopify_timezone variable, which converts ALL timestamps included in the package (including _fivetran_synced) to a single target timezone in IANA Database format, ie "America/Los_Angeles" (PR #41).
• shopify_<default_source_table_name>_identifier variables added if an individual source table has a different name than the package expects (PR #38).
• The declaration of passthrough variables within your root dbt_project.yml has changed (but is backwards compatible). To allow for more flexibility and better tracking of passthrough columns, you will now want to define passthrough columns in the following format (PR #40):

This applies to all passthrough columns within the dbt_shopify_source package and not just the customer_pass_through_columns example. See the README for which models have passthrough columns.

vars:
  customer_pass_through_columns:
    - name: "my_field_to_include" # Required: Name of the field within the source.
      alias: "field_alias" # Optional: If you wish to alias the field within the staging model.
      transform_sql: "cast(field_alias as string)" # Optional: If you wish to define the datatype or apply a light transformation.

• The following fields have been added to (➕) or removed from (➖) their respective staging models (PR #39, PR #40):
  • stg_shopify__order:
    • ➕ total_discounts_set
    • ➕ total_line_items_price_set
    • ➕ total_price_usd
    • ➕ total_price_set
    • ➕ total_tax_set
    • ➕ total_tip_received
    • ➕ is_deleted
    • ➕ app_id
    • ➕ checkout_id
    • ➕ client_details_user_agent
    • ➕ customer_locale
    • ➕ order_status_url
    • ➕ presentment_currency
    • ➕ is_confirmed
  • stg_shopify__customer:
    • ➕ note
    • ➕ lifetime_duration
    • ➕ currency
    • ➕ marketing_consent_state (coalescing of email_marketing_consent_state and deprecated accepts_marketing field)
    • ➕ marketing_opt_in_level (coalescing of email_marketing_consent_opt_in_level and deprecated marketing_opt_in_level field)
    • ➕ marketing_consent_updated_at (coalescing of email_marketing_consent_consent_updated_at and deprecated accepts_marketing_updated_at field)
    • ➖ accepts_marketing/has_accepted_marketing
    • ➖ accepts_marketing_updated_at
    • ➖ marketing_opt_in_level
  • stg_shopify__order_line_refund:
    • ➕ subtotal_set
    • ➕ total_tax_set
  • stg_shopify__order_line:
    • ➕ pre_tax_price_set
    • ➕ price_set
    • ➕ tax_code
    • ➕ total_discount_set
    • ➕ variant_title
    • ➕ variant_inventory_management
    • ➕ properties
    • ( ) is_requiring_shipping is renamed to is_shipping_required
  • stg_shopify__product:
    • ➕ status
  • stg_shopify__product_variant
    • ➖ old_inventory_quantity -> coalesced with inventory_quantity
    • ➕ inventory_quantity -> coalesced with old_inventory_quantity
• The following source tables have been added to the package with respective staging models (PR #39):
  • abandoned_checkout
  • collection_product
  • collection
  • customer_tag
  • discount_code -> if the table does not exist in your schema, the package will create an empty staging model and reference that (PR #47, see DECISIONLOG)
  • fulfillment
  • inventory_item
  • inventory_level
  • location
  • metafield (#PR 49 as well)
  • order_note_attribute
  • order_shipping_line
  • order_shipping_tax_line
  • order_tag
  • order_url_tag
  • price_rule
  • product_image
  • product_tag
  • shop
  • tender_transaction
  • abandoned_checkout_discount_code
  • order_discount_code
  • tax_line
  • abandoned_checkout_shipping_line ((PR #47) as well)
  • fulfillment_event -> This is NOT included by default. To include fulfillment events (used in the shopify__daily_shop model), set the shopify_using_fulfillment_event variable to true (PR #48)