add callout for net new snapshots #6521
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
github-actions
bot
added
content
Co-authored-by: Grace Goheen <[email protected]>
thanks @graciegoheen , i did meet with @dbeatty10 last week and thought he meant the new yaml spec. will loop Doug to confirm to make sure before i make further changes. |
| @@ -150,11 +150,27 @@ Configurations can be applied to snapshots using the [YAML syntax](/docs/build/s | |||
|
|
|||
| </Tabs> | |||
|
|
|||
| ### Snapshot configuration migration | |||
|
|
|||
| The latest snapshot YAML configurations introduced in dbt v1.9 and higher (such as [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) and [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current)) are best suited for new snapshots. For existing snapshots, we recommend the following to avoid any inconsistencies to your snapshots: | |||
There was a problem hiding this comment.
note i'm using 'latest' so it's more scalable for the long term. if i use 'new', it'll become outdated eventually i think.
Co-authored-by: Matt Shaver <[email protected]>
Co-authored-by: Matt Shaver <[email protected]>
Co-authored-by: Matt Shaver <[email protected]>
Co-authored-by: Matt Shaver <[email protected]>
| [+](/reference/resource-configs/plus-prefix)[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>} | ||
| [+](/reference/resource-configs/plus-prefix)[invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) : true | false | ||
| [+](/reference/resource-configs/plus-prefix)[dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current): <string> |
There was a problem hiding this comment.
dbt_valid_to_current config is available in 1.9+
There was a problem hiding this comment.
Oh maybe I'm not reading these version blocks right?
There was a problem hiding this comment.
@graciegoheen You gotta expand the view so you can see line 67 / 64:
| @@ -150,11 +150,27 @@ Configurations can be applied to snapshots using the [YAML syntax](/docs/build/s | |||
|
|
|||
| </Tabs> | |||
|
|
|||
| ### Snapshot configuration migration | |||
|
|
|||
| The latest snapshot YAML configurations introduced in dbt v1.9 (such as [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) and [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current)) are best suited for new snapshots. For existing snapshots, we recommend the following to avoid any inconsistencies in your snapshots: | |||
There was a problem hiding this comment.
I think it's a short enough list we can just name them all - snapshot_meta_column_names, dbt_valid_to_current, hard_deletes
There was a problem hiding this comment.
Also, you say "For existing snapshots, we recommend the following" but then go into a list for new snapshots?
There was a problem hiding this comment.
good call! I'll swap so it flows better 🙏
| @@ -150,11 +150,27 @@ Configurations can be applied to snapshots using the [YAML syntax](/docs/build/s | |||
|
|
|||
| </Tabs> | |||
|
|
|||
| ### Snapshot configuration migration | |||
|
|
|||
| The latest snapshot YAML configurations introduced in dbt v1.9 (such as [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) and [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current)) are best suited for new snapshots. For existing snapshots, we recommend the following to avoid any inconsistencies in your snapshots: | |||
There was a problem hiding this comment.
| The latest snapshot YAML configurations introduced in dbt v1.9 (such as [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) and [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current)) are best suited for new snapshots. For existing snapshots, we recommend the following to avoid any inconsistencies in your snapshots: | |
| The latest snapshot configurations introduced in dbt v1.9 (such as [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) and [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current)) are best suited for new snapshots. For existing snapshots, we recommend the following to avoid any inconsistencies in your snapshots: |
| #### For new snapshots | ||
| - Use the latest snapshot YAML configurations when creating new snapshots that didn't exist previously. |
There was a problem hiding this comment.
I think we can just delete this?
| - Migrate tables — Migrate the previous snapshot to the new table schema and values: | ||
| - Create a backup copy of your snapshots. | ||
| - Use `alter` statements as needed (or a script to apply `alter` statements) to ensure table consistency. | ||
| - New YAML config — Convert the YAML configurations one at a time, testing as you go. |
There was a problem hiding this comment.
We can just call these "new configs" we don't need to say YAML configs
| - Use `alter` statements as needed (or a script to apply `alter` statements) to ensure table consistency. | ||
| - New YAML config — Convert the YAML configurations one at a time, testing as you go. | ||
|
|
||
| If you use one of the latest configs, such as `dbt_valid_to_current`, without migrating your data, you may have mixed old and new data, leading to an incorrect downstream result. |
There was a problem hiding this comment.
should this be in a warning callout?
| @@ -0,0 +1 @@ | |||
| Define snapshots using the latest [YAML configurations](/docs/build/snapshots#configuring-snapshots), available in Versionless dbt Cloud and Core v1.9 and higher, in the `snapshot` directory file. | |||
There was a problem hiding this comment.
is this about the YAML syntax?
I think we're getting two separate things muddied:
- starting in 1.9 you can define snapshots in yaml
- starting in 1.9 there are a handful of new configs that you can use, we recommend only using those configs on new snapshots or if you use them on old snapshots you will need to migrate
Adding a callout to snapshot config page to clarify the new yaml config for net new snapshots. exisitng config snapshots can continue using config block, however added guidance on what to do if they want to migrate to new yaml spec.
Resolves #6374
🚀 Deployment available! Here are the direct links to the updated files: