Refactor pytest unit tests to dbt unit tests

.github/workflows/run_unit_tests_on_pr.yml

@@ -3,6 +3,9 @@ name: Run Unit Tests on Pull Request
 on: [pull_request_target,workflow_dispatch]
 env:
   BIGQUERY_PROJECT: ${{ secrets.BIGQUERY_PROJECT }}
+  BIGQUERY_PROPERTY_ID: ${{ secrets.BIGQUERY_PROPERTY_ID }}
+  BIGQUERY_DATASET: ${{ secrets.BIGQUERY_DATASET }}
+  BIGQUERY_KEYFILE: ./unit_tests/dbt-service-account.json
 
 jobs:
   pytest_run_all:
@@ -35,3 +38,36 @@ jobs:
 
       - name: Run tests
         run: python -m pytest .
+
+  run_dbt_unit_tests:
+    name: Run dbt Unit Tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out
+        uses: actions/checkout@v3
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+
+      - uses: actions/setup-python@v1
+        with:
+          python-version: "3.11.x"
+
+      - name: Authenticate using service account
+        run: 'echo "$KEYFILE" > ./unit_tests/dbt-service-account.json'
+        shell: bash
+        env:
+          KEYFILE: ${{ secrets.GCP_BIGQUERY_USER_KEYFILE }}
+
+      - name: Install dbt
+        run: |
+          pip install dbt-core
+          pip install dbt-bigquery
+          dbt deps
+
+      - name: Materialize necessary dbt resources
+        run: |
+          dbt seed -f
+          dbt run -s +test_type:unit -f --empty
+
+      - name: Run dbt unit tests
+        run: dbt test -s test_type:unit

.gitignore

@@ -2,6 +2,8 @@
 target/
 dbt_packages/
 logs/
+package-lock.yml
+.user.yml
 
 google-cloud-sdk/
 unit_tests/.env

README.md

@@ -72,7 +72,7 @@ packages:
 ```
 ## Required Variables
 
-This package assumes that you have an existing DBT project with a BigQuery profile and a BigQuery GCP instance available with GA4 event data loaded. Source data is defined using the `project` and `dataset` variables below. The `static_incremental_days` variable defines how many days' worth of data to reprocess during incremental runs. 
+This package assumes that you have an existing DBT project with a BigQuery profile and a BigQuery GCP instance available with GA4 event data loaded. Source data is defined using the `project` and `property_ids` variables below. The `static_incremental_days` variable defines how many days' worth of data to reprocess during incremental runs. The `start_date` variable defines the earliest date for which data is included and loaded into the models in this package.
 
 ```
 vars:
@@ -214,6 +214,8 @@ vars:
         value_type: "string_value"
 ```
 
+The `derived_user_properties` set in `dbt_project.yml` should either be updated to reflect the derived user properties for your project or they should be removed if you don't wish to set any derived user properties.
+
 ### Derived Session Properties
 
 Derived session properties are similar to derived user properties, but on a per-session basis, for properties that change slowly over time. This provides additional flexibility in allowing users to turn any event parameter into a session property. 
@@ -247,6 +249,8 @@ vars:
         value_type: "int_value"
 ```
 
+The `derived_session_properties` set in `dbt_project.yml` should either be updated to reflect the derived session properties for your project or they should be removed if you don't wish to set any derived session properties.
+
 ### GA4 Recommended Events
 
 See the README file at /dbt_packages/models/staging/recommended_events for instructions on enabling [Google's recommended events](https://support.google.com/analytics/answer/9267735?hl=en).
@@ -261,6 +265,8 @@ vars:
     conversion_events: ['purchase','download']
 ```
 
+The `conversion_events` set in `dbt_project.yml` should either be updated to reflect the conversion events for your project or they should be removed if you don't wish to set any conversion events.
+
 ### Session Attribution Lookback Window
 
 The `stg_ga4__sessions_traffic_sources_last_non_direct_daily` model provides last non-direct session attribution within a configurable lookback window. The default is 30 days, but this can be overridden with the `session_attribution_lookback_window_days` variable.
@@ -302,9 +308,35 @@ The easiest option is using OAuth with your Google Account. Summarized instructi
 ```
 gcloud auth application-default login --scopes=https://www.googleapis.com/auth/bigquery,https://www.googleapis.com/auth/iam.test
 ```
+
+The `profiles.yml` file included in this package should be removed. The `profile: 'default'` line in `dbt_project.yml` in this package should also be removed.
+
 # Unit Testing
 
-This package uses `pytest` as a method of unit testing individual models. More details can be found in the [unit_tests/README.md](unit_tests) folder.
+The dbt-ga4 package treats each model and macro as a 'unit' of code. If we fix the input to each unit, we can test that we received the expected output.
+
+This package currently uses a combination of dbt unit tests and `pytest` as a method of unit testing individual models. The remaining `pytest` unit test will be refactored to a dbt unit test when possible - progress on the bug preventing that work can be tracked [here](https://github.com/dbt-labs/dbt-core/issues/10353).
+
+### dbt unit tests
+
+dbt's documentation on unit tests can be found [here](https://docs.getdbt.com/docs/build/unit-tests). Unit tests are performed the same way other types of dbt tests are executed.
+
+Execute a specific test:
+```
+dbt test -s <test_name>
+```
+Execute all tests configured for a model:
+```
+dbt test -s <model_name>
+```
+Execute all dbt unit tests:
+```
+dbt test -s test_type:unit
+```
+
+### pytest
+
+More details on using `pytest` for unit testing can be found in the [unit_tests/README.md](unit_tests) folder.
 
 # Overriding Default Channel Groupings
 

TODO.md

@@ -22,6 +22,7 @@
 - Configuration and dynamic templates to create custom event tables and dimensions
 - Configuration to create custom dimensions (session, user, event_*) from event parameters
 - Use Fivetran's `union_data` method (or something similar) to handle multiple, unioned GA4 exports. https://github.com/fivetran/dbt_xero_source/blob/main/models/tmp/stg_xero__account_tmp.sql
+- Un-comment unit test in `stg_ga4__sessions_traffic_sources_last_non_direct_daily.yml` once [this bug](https://github.com/dbt-labs/dbt-core/issues/10353) is resolved. Once that is complete, the `unit_tests` folder pertaining to the `pytest` unit tests should be removed along with the `pytest_run_all` job in `run_unit_tests_on_pr.yml`.
 
 ## Misc
 

dbt_project.yml

@@ -8,6 +8,23 @@ seed-paths: ["seeds"]
 macro-paths: ["macros"]
 snapshot-paths: ["snapshots"]
 
+profile: 'default'
+
+# These variables are used for unit tests during CI for the package
+# These variables should either be removed or updated to reflect the needs of your GA data and dbt project
+vars:
+  start_date: "20230306"
+  static_incremental_days: 3
+  derived_session_properties:
+    - event_parameter: "page_location"
+      session_property_name: "most_recent_page_location"
+      value_type: "string_value"
+  derived_user_properties:
+    - event_parameter: "page_title"
+      user_property_name: "most_recent_page_title"
+      value_type: "string_value"
+  conversion_events: ['large_button_clicked', 'add_to_cart']
+
 target-path: "target"  # directory which will store compiled SQL files
 clean-targets:         # directories to be removed by `dbt clean`
   - "target"

macros/base_select.sql

@@ -36,7 +36,8 @@
     , ecommerce.transaction_id
     , items
     , {%- if  var('combined_dataset', false) != false %} cast(left(regexp_replace(_table_suffix, r'^(intraday_)?\d{8}', ''), 100) as int64)
-        {%- else %} {{ var('property_ids')[0] }}
+        {%- elif var('property_ids', false) != false %} {{ var('property_ids')[0] }}
+        {%- else %} {{ env_var('BIGQUERY_PROPERTY_ID') }}
         {%- endif %} as property_id
 {% endmacro %}
 

models/staging/base/base_ga4__events.sql

@@ -21,9 +21,11 @@ with source as (
     select
         {{ ga4.base_select_source() }}
     from {{ source('ga4', 'events') }}
-    where cast(left(replace(_table_suffix, 'intraday_', ''), 8) as int64) >= {{var('start_date')}}
-    {% if is_incremental() %}
-        and parse_date('%Y%m%d', left(replace(_table_suffix, 'intraday_', ''), 8)) in ({{ partitions_to_replace | join(',') }})
+    {% if not flags.EMPTY %}
+        where cast(left(replace(_table_suffix, 'intraday_', ''), 8) as int64) >= {{var('start_date')}}
+        {% if is_incremental() %}
+            and parse_date('%Y%m%d', left(replace(_table_suffix, 'intraday_', ''), 8)) in ({{ partitions_to_replace | join(',') }})
+        {% endif %}
     {% endif %}
 ),
 renamed as (

models/staging/src_ga4.yml

@@ -4,11 +4,13 @@ sources:
   - name: ga4
     database: | # Source from target.project if multi-property, otherwise source from source_project
       {%- if  var('combined_dataset', false) != false -%} {{target.project}}
-      {%- else -%} {{var('source_project')}}
+      {%- elif var('source_project', false) != false -%} {{var('source_project')}}
+      {%- else -%} {{env_var('BIGQUERY_PROJECT')}}
       {%- endif -%}
     schema: | # Source from combined property dataset if set, otherwise source from original GA4 property
       {%- if  var('combined_dataset', false) != false -%} {{var('combined_dataset')}}
-      {%- else -%} analytics_{{var('property_ids')[0]}}
+      {%- elif var('property_ids', false) != false -%} analytics_{{var('property_ids')[0]}}
+      {%- else -%} analytics_{{env_var('BIGQUERY_PROPERTY_ID')}}
       {%- endif -%}
     tables:
       - name: events

models/staging/stg_ga4__client_key_first_last_events.yml

@@ -7,4 +7,20 @@ models:
       - name: client_key
         description: Hashed combination of user_pseudo_id and stream_id
         tests: 
-          - unique
+          - unique
+unit_tests:
+  - name: test_stg_ga4__client_key_first_last_events
+    description: Test pulling the first and last event per client key
+    model: stg_ga4__client_key_first_last_events
+    given:
+      - input: ref('stg_ga4__events')
+        format: csv
+        rows: |
+          stream_id,client_key,event_key,event_timestamp
+          1,IX+OyYJBgjwqML19GB/XIQ==,H06dLW6OhNJJ6SoEPFsSyg==,1661339279816517
+          1,IX+OyYJBgjwqML19GB/XIQ==,gt1SoAtrxDv33uDGwVeMVA==,1661339279816518
+    expect:
+      format: csv
+      rows: |
+        client_key,first_event,last_event
+        IX+OyYJBgjwqML19GB/XIQ==,H06dLW6OhNJJ6SoEPFsSyg==,gt1SoAtrxDv33uDGwVeMVA==

models/staging/stg_ga4__client_key_first_last_pageviews.yml

@@ -7,4 +7,20 @@ models:
       - name: client_key
         description: Hashed combination of user_pseudo_id and stream_id
         tests:
-          - unique
+          - unique
+unit_tests:
+  - name: test_stg_ga4__client_key_first_last_pageviews
+    description: Test pulling the first and last page view per client key
+    model: stg_ga4__client_key_first_last_pageviews
+    given:
+      - input: ref('stg_ga4__event_page_view')
+        format: csv
+        rows: |
+          stream_id,client_key,event_key,event_timestamp,page_location
+          1,IX+OyYJBgjwqML19GB/XIQ==,H06dLW6OhNJJ6SoEPFsSyg==,1661339279816517,A
+          1,IX+OyYJBgjwqML19GB/XIQ==,gt1SoAtrxDv33uDGwVeMVA==,1661339279816518,B
+    expect:
+      format: csv
+      rows: |
+        client_key,first_page_view_event_key,last_page_view_event_key,first_page_location,last_page_location
+        IX+OyYJBgjwqML19GB/XIQ==,H06dLW6OhNJJ6SoEPFsSyg==,gt1SoAtrxDv33uDGwVeMVA==,A,B

models/staging/stg_ga4__derived_session_properties.yml

@@ -8,4 +8,39 @@ models:
     columns:
       - name: session_key
         tests:
-          - unique  
+          - unique
+unit_tests:
+  - name: test_derived_session_properties
+    description: Test whether a derived property is successfully retrieved from multiple event payloads
+    model: stg_ga4__derived_session_properties
+    given:
+      - input: ref('stg_ga4__events')
+        format: sql
+        rows: |
+          select 
+          'AAA' as session_key
+          , 1617691790431476 as event_timestamp
+          , 'first_visit' as event_name
+          , ARRAY[STRUCT('my_param' as key, STRUCT(1 as int_value) as value)] as event_params
+          , ARRAY[STRUCT('my_property' as key, STRUCT('value1' as string_value) as value)] as user_properties
+          union all
+          select
+          'AAA' as session_key
+          , 1617691790431477 as event_timestamp
+          , 'first_visit' as event_name
+          , ARRAY[STRUCT('my_param' as key, STRUCT(2 as int_value) as value)] as event_params
+          , ARRAY[] as user_properties
+          union all
+          select 
+          'BBB' as session_key
+          , 1617691790431477 as event_timestamp
+          , 'first_visit' as event_name
+          , ARRAY[STRUCT('my_param' as key, STRUCT(1 as int_value) as value)] as event_params
+          , ARRAY[STRUCT('my_property' as key, STRUCT('value2' as string_value) as value)] as user_properties
+    expect:
+      format: dict
+      rows:
+        - {session_key: AAA, my_derived_property: 2, my_derived_property2: value1}
+        - {session_key: BBB, my_derived_property: 1, my_derived_property2: value2}
+    overrides:
+      vars: {derived_session_properties: [{event_parameter: 'my_param',session_property_name: 'my_derived_property',value_type: 'int_value'},{user_property: 'my_property',session_property_name: 'my_derived_property2',value_type: 'string_value'}]}

models/staging/stg_ga4__derived_user_properties.yml

@@ -7,4 +7,36 @@ models:
       - name: client_key
         description: Hashed combination of user_pseudo_id and stream_id
         tests:
-          - unique  
+          - unique  
+unit_tests:
+  - name: test_derived_user_properties
+    description: Test whether a derived user property is successfully retrieved from multiple event payloads
+    model: stg_ga4__derived_user_properties
+    given:
+      - input: ref('stg_ga4__events')
+        format: sql
+        rows: |
+          select 
+          'AAA' as client_key
+          , 1617691790431476 as event_timestamp
+          , 'first_visit' as event_name
+          , ARRAY[STRUCT('my_param' as key, STRUCT(1 as int_value) as value)] as event_params
+          union all
+          select
+          'AAA' as client_key
+          , 1617691790431477 as event_timestamp
+          , 'first_visit' as event_name
+          , ARRAY[STRUCT('my_param' as key, STRUCT(2 as int_value) as value)] as event_params
+          union all
+          select 
+          'BBB' as client_key
+          , 1617691790431477 as event_timestamp
+          , 'first_visit' as event_name
+          , ARRAY[STRUCT('my_param' as key, STRUCT(1 as int_value) as value)] as event_params
+    expect:
+      format: dict
+      rows:
+        - {client_key: AAA, my_derived_property: 2}
+        - {client_key: BBB, my_derived_property: 1}
+    overrides:
+      vars: {derived_user_properties: [{event_parameter: 'my_param',user_property_name: 'my_derived_property',value_type: 'int_value'}]}

models/staging/stg_ga4__event_to_query_string_params.yml

@@ -3,4 +3,23 @@ version: 2
 models:  
   - name: stg_ga4__event_to_query_string_params
     description: This model pivots the query string parameters contained within the event's page_location field to become rows. Each row is a single parameter/value combination contained in a single event's query string.
-
+unit_tests:
+  - name: test_stg_ga4__event_to_query_string_params
+    description: Test whether event query strings are flattened for each query string parameter
+    model: stg_ga4__event_to_query_string_params
+    given:
+      - input: ref('stg_ga4__events')
+        format: csv
+        rows: |
+          event_key,page_query_string
+          aaa,param1=value1&param2=value2
+          bbb,param1
+          ccc,param1=
+    expect:
+      format: csv
+      rows: |
+        event_key,param,value
+        aaa,param1,value1
+        aaa,param2,value2
+        bbb,param1,
+        ccc,param1,