-
Notifications
You must be signed in to change notification settings - Fork 959
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add callout for net new snapshots #6521
base: current
Are you sure you want to change the base?
Changes from 20 commits
befa7b5
0697083
0ff5771
4127104
5a736c5
6746196
64d56e2
0347fd3
576af99
2532cad
472d5c0
eee70d8
02fd890
1ca7c4e
43386c6
832ef12
91c3bb9
14690e0
7e68e91
6d2a38d
38762c1
bdf2850
6a84a7f
37efd7a
7eb3bf1
2fba6cc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,21 +7,18 @@ | |
|
||
import ConfigResource from '/snippets/_config-description-resource.md'; | ||
import ConfigGeneral from '/snippets/_config-description-general.md'; | ||
import SnapshotsYamlConfig from '/snippets/_snapshots-yaml-config.md'; | ||
|
||
|
||
## Related documentation | ||
* [Snapshots](/docs/build/snapshots) | ||
* The `dbt snapshot` [command](/reference/commands/snapshot) | ||
|
||
<!-- | ||
Parts of a snapshot: | ||
- name | ||
- query | ||
--> | ||
|
||
## Available configurations | ||
### Snapshot-specific configurations | ||
|
||
|
||
<ConfigResource meta={frontMatter.meta} /> | ||
|
||
<VersionBlock lastVersion="1.8"> | ||
|
@@ -78,8 +75,9 @@ | |
[+](/reference/resource-configs/plus-prefix)[strategy](/reference/resource-configs/strategy): timestamp | check | ||
[+](/reference/resource-configs/plus-prefix)[updated_at](/reference/resource-configs/updated_at): <column_name> | ||
[+](/reference/resource-configs/plus-prefix)[check_cols](/reference/resource-configs/check_cols): [<column_name>] | all | ||
[+](/reference/resource-configs/plus-prefix)[invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) : true | false | ||
[+](/reference/resource-configs/plus-prefix)[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>} | ||
[+](/reference/resource-configs/plus-prefix)[invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) : true | false | ||
[+](/reference/resource-configs/plus-prefix)[dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current): <string> | ||
``` | ||
|
||
</File> | ||
|
@@ -112,8 +110,9 @@ | |
[strategy](/reference/resource-configs/strategy): timestamp | check | ||
[updated_at](/reference/resource-configs/updated_at): <column_name> | ||
[check_cols](/reference/resource-configs/check_cols): [<column_name>] | all | ||
[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>} | ||
[invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) : true | false | ||
[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>} | ||
[dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current): <string> | ||
``` | ||
</File> | ||
|
||
|
@@ -125,7 +124,8 @@ | |
|
||
<VersionBlock firstVersion="1.9"> | ||
|
||
Configurations can be applied to snapshots using the [YAML syntax](/docs/build/snapshots), available in Versionless and dbt v1.9 and higher, in the `snapshot` directory file. | ||
<SnapshotsYamlConfig /> | ||
|
||
|
||
</VersionBlock> | ||
|
||
|
@@ -150,11 +150,27 @@ | |
|
||
</Tabs> | ||
|
||
### Snapshot configuration migration | ||
|
||
The latest snapshot YAML configurations introduced in dbt v1.9 and higher (such as [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) and [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current)) are best suited for new snapshots. For existing snapshots, we recommend the following to avoid any inconsistencies to your snapshots: | ||
Check warning on line 155 in website/docs/reference/snapshot-configs.md GitHub Actions / vale[vale] website/docs/reference/snapshot-configs.md#L155
Raw output
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. note i'm using 'latest' so it's more scalable for the long term. if i use 'new', it'll become outdated eventually i think.
mirnawong1 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
#### For new snapshots | ||
- Use the latest snapshot YAML configurations when creating new snapshots that didn't exist previously. | ||
Comment on lines
+157
to
+158
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we can just delete this? |
||
|
||
#### For existing snapshots | ||
- Migrate tables — Migrate the previous snapshot to the new table schema and values: | ||
- Create a backup copy of your snapshots. | ||
- Use `alter` statements as needed to ensure table consistency or use a script to apply the `alter` statements. | ||
mirnawong1 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- New YAML config — Convert the YAML configurations one at a time, testing as you go. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can just call these "new configs" we don't need to say YAML configs |
||
|
||
If you use one of the latest configs, such as `dbt_valid_to_current`, without migrating your data may result in mixed old and new data, leading to incorrect downstream result. | ||
mirnawong1 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
### General configurations | ||
|
||
<ConfigGeneral /> | ||
|
||
|
||
<Tabs | ||
groupId="config-languages" | ||
defaultValue="project-yaml" | ||
|
@@ -170,6 +186,7 @@ | |
|
||
<VersionBlock firstVersion="1.9"> | ||
|
||
|
||
```yaml | ||
snapshots: | ||
[<resource-path>](/reference/resource-configs/resource-path): | ||
|
@@ -256,7 +273,7 @@ | |
|
||
<VersionBlock firstVersion="1.9"> | ||
|
||
Configurations can be applied to snapshots using [YAML syntax](/docs/build/snapshots), available in Versionless and dbt v1.9 and higher, in the `snapshot` directory file. | ||
<SnapshotsYamlConfig /> | ||
|
||
</VersionBlock> | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Define snapshots using the latest [YAML configurations](/docs/build/snapshots#configuring-snapshots), available in Versionless and dbt v1.9 and higher, in the `snapshot` directory file. | ||
Check warning on line 1 in website/snippets/_snapshots-yaml-config.md GitHub Actions / vale[vale] website/snippets/_snapshots-yaml-config.md#L1
Raw output
|
||
mirnawong1 marked this conversation as resolved.
Show resolved
Hide resolved
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -145,4 +145,5 @@ | |
.headerText { | ||
display: flex; | ||
align-items: center; | ||
} | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dbt_valid_to_current
config is available in 1.9+There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh maybe I'm not reading these version blocks right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@graciegoheen You gotta expand the view so you can see line 67 / 64: