Skip to content

Commit

Permalink
chore: document airbyte omie historical price hour change from fr to …
Browse files Browse the repository at this point in the history
…incremental (et/somenergia-jardiner!27)

* chore: document airbyte omie historical price hour change from fr to incremental
  • Loading branch information
polmonso committed Nov 17, 2023
1 parent 1dad998 commit 016c557
Show file tree
Hide file tree
Showing 2 changed files with 17 additions and 29 deletions.
3 changes: 3 additions & 0 deletions dbt_jardiner/models/jardiner/raw/omie/_raw_omie__sources.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,6 @@ sources:
- name: omie_historical_price_hour
description: >
Taula importada via airbyte del projecte de KPIs antic.
airbyte currently normalizes the data (JSONB -> table). To avoid drop cascade we switched to incremental
Alternativelly, we can full-refresh without normalization since _airbyte_raw_* is truncated, not replaced.
https://discuss.airbyte.io/t/remove-related-VIEWS-IN-full-refresh-overwrite-MODE/747
Original file line number Diff line number Diff line change
@@ -1,31 +1,16 @@
{{ config(
materialized = 'view'
) }}
{#
todo airbyte IS currently doing A full - refresh which DROP - cascades,
we materialize this model TO avoid checking if we can incremental - sync instead but it would NOT happen if we don 'T normalize the data, which we don' t actually need TO DO https:// discuss.airbyte.io / t / remove - related - VIEWS - IN - full - refresh - overwrite - MODE / 747 #}
WITH recent AS (
{{ config(materialized="view") }}

SELECT
"date",
create_time,
price,
ROW_NUMBER() OVER (
PARTITION BY "date"
ORDER BY
create_time DESC
) AS rc
FROM
{{ source(
'plantlake',
'omie_historical_price_hour'
) }}
)
with
recent as (

SELECT
"date" AS start_hour,
price
FROM
recent
WHERE
rc = 1
select
"date",
create_time,
price,
row_number() over (partition by "date" order by create_time desc) as rc
from {{ source("plantlake", "omie_historical_price_hour") }}
)

select "date" as start_hour, price
from recent
where rc = 1

0 comments on commit 016c557

Please sign in to comment.