Skip to content

Commit

Permalink
chore(pipeline): cleanup model configs
Browse files Browse the repository at this point in the history
  • Loading branch information
vmttn committed Sep 19, 2024
1 parent a4bcc01 commit a28d52f
Showing 1 changed file with 20 additions and 47 deletions.
67 changes: 20 additions & 47 deletions pipeline/dbt/models/intermediate/_models.yml
Original file line number Diff line number Diff line change
@@ -1,25 +1,18 @@
version: 2

x-union-common-check-args: &union-common-check-args
include:
- _di_surrogate_id
config:
severity: warn
store_failures: true

models:
- name: int__union_adresses
description: |
Gathers addresses from all sources
- name: int__plausible_personal_emails

* model can contain faulty data
* test failure are saved (see log output)
- name: int__union_adresses
data_tests:
- check_adresse: *union-common-check-args
- check_adresse:
include:
- _di_surrogate_id
config:
severity: warn
store_failures: true

- name: int__union_contacts
description: |
Gathers contacts from all sources
columns:
- name: contact_uid
data_tests:
Expand All @@ -31,38 +24,26 @@ models:
- dbt_utils.not_empty_string

- name: int__union_services
description: |
Gathers services from all sources
* model can contain faulty data
* test failure are saved (see log output)
data_tests:
- check_service: *union-common-check-args
- check_service:
include:
- _di_surrogate_id
config:
severity: warn
store_failures: true

- name: int__union_structures
description: |
Gathers structures from all sources
* model can contain faulty data
* test failure are saved (see log output)
data_tests:
- check_structure: *union-common-check-args

- name: int__plausible_personal_emails
- check_structure:
include:
- _di_surrogate_id
config:
severity: warn
store_failures: true

- name: int__union_adresses__enhanced
description: |
All valid adresses, with geocoding

- name: int__union_services__enhanced
description: |
All valid services, with extra data:
* geocoded addresses
* zone_diffusion_* filled with geocoded data (monenfant, soliguide)
A service belonging to a structure data failing validation is considered invalid.
columns:
- name: zone_diffusion_code
data_tests:
Expand Down Expand Up @@ -99,19 +80,11 @@ models:
severity: warn

- name: int__union_structures__enhanced
description: |
All valid structures, with extra data:
* geocoded addresses
* email with pii flag

- name: int__geocodages
description: |
Geocoding results for all sources.
This model is incremental, it will only geocode new or changed addresses.
It stores raw geocoding results, without filtering.
Geocoding is done by calling the BAN api in PL/Python.
columns:
- name: geocoded_at
Expand Down

0 comments on commit a28d52f

Please sign in to comment.