Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] 1.9 upgrade guide #6184

Merged
merged 37 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
eb6e590
adding upgrade guide
runleonarun Oct 1, 2024
179e624
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 1, 2024
5cf5f67
updating with internal changes
runleonarun Oct 1, 2024
77b743d
fixing link
runleonarun Oct 1, 2024
438f5a9
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 1, 2024
26abb99
Update website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1…
runleonarun Oct 1, 2024
85bda08
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 1, 2024
a6e475c
Update website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1…
runleonarun Oct 1, 2024
8c8eff1
fixing link
runleonarun Oct 1, 2024
18bcb9f
adding adapters
runleonarun Oct 1, 2024
d49c1f7
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 1, 2024
00462f2
Apply suggestions from code review
runleonarun Oct 1, 2024
d2143e7
removing INSERT link
runleonarun Oct 2, 2024
7a5391d
Update website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1…
amychen1776 Oct 2, 2024
932874b
Apply suggestions from code review
runleonarun Oct 2, 2024
1365f9d
Update website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1…
runleonarun Oct 2, 2024
ba06ec2
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 2, 2024
56c362b
Apply suggestions from code review
runleonarun Oct 2, 2024
3d45b3f
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 2, 2024
e7a2d46
Apply suggestions from code review
runleonarun Oct 2, 2024
4c6bf6c
Apply suggestions from code review
runleonarun Oct 2, 2024
74f519c
Update 06-upgrading-to-v1.9.md
runleonarun Oct 2, 2024
866a0fe
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 2, 2024
fef4db3
@grace feedback
runleonarun Oct 2, 2024
24b0dd5
Apply suggestions from code review
runleonarun Oct 2, 2024
28cd2c7
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 2, 2024
a6b0a62
Remove `state_modified_compare_vars` from the upgrade guide
dbeatty10 Oct 2, 2024
33628eb
Update website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1…
runleonarun Oct 2, 2024
f50f2a4
Merge branch 'current' into upgrade-1.9-guide
mirnawong1 Oct 3, 2024
9bbcce4
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 3, 2024
74a30ef
Apply suggestions from code review
runleonarun Oct 3, 2024
c7b2e20
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 3, 2024
5d8ae6e
Update website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1…
runleonarun Oct 3, 2024
06bbe85
Update 06-upgrading-to-v1.9.md
runleonarun Oct 3, 2024
5613080
Update 06-upgrading-to-v1.9.md
runleonarun Oct 3, 2024
1ae0f55
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 3, 2024
f58be81
Merge branch 'current' into upgrade-1.9-guide
runleonarun Oct 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ exports.versions = [
version: "1.9.1",
customDisplay: "Cloud (Versionless)",
},
{
version: "1.9",
isPrerelease: true,
},
{
version: "1.8",
EOLDate: "2025-04-15",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
---
title: "Upgrading to v1.9 (beta)"
id: upgrading-to-v1.9
description: New features and changes in dbt Core v1.9
displayed_sidebar: "docs"
---

graciegoheen marked this conversation as resolved.
Show resolved Hide resolved
## Resources

- Changelog INSERT HERE - LINK to 1.9 changelog
runleonarun marked this conversation as resolved.
Show resolved Hide resolved
- [dbt Core CLI Installation guide](/docs/core/installation-overview)
- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud)
runleonarun marked this conversation as resolved.
Show resolved Hide resolved

## What to know before upgrading

dbt Labs is committed to providing backward compatibility for all versions 1.x, except for any changes explicitly mentioned on this page. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new).

Remember from version 1.8 that we're [going versionless](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8#versionless) and we have a new [adapter installation procedure](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8#new-dbt-core-adapter-installation-procedure).
runleonarun marked this conversation as resolved.
Show resolved Hide resolved

## New and changed features and functionality

Features and functionality new in dbt v1.9.

### New microbatch `incremental_strategy`
runleonarun marked this conversation as resolved.
Show resolved Hide resolved

INSERT HERE - link to docs
runleonarun marked this conversation as resolved.
Show resolved Hide resolved

Historically, managing incremental models involved several manual steps and responsibilities, which involved using:

* Explicit filtering to define "new" data by writing your SQL within an `is_incremental` block.
* Custom logic for incremental loads by implementing your own logic to handle different loading strategies, such as `append` or `delete+insert`.
* Handle batches manually by implementing custom logic using variables.

These steps made the process error-prone and introduced performance concerns because you had to run a single large SQL query to process all new and updated records.

Starting in Core 1.9, you can use the new microbatch strategy, which streamlines this process and automates many of these tasks. The benefits include:

* Simplified query design: Write your model query for a single day of data and no longer need `is_incremental()` logic or manual SQL for determining "new" records.
* Automatic batch processing: dbt automatically breaks down the loading process into smaller batches based on the specified `batch_size` and handles the SQL queries for each batch independently, improving efficiency and reducing the risk of query timeouts.
* Dynamic filtering: Use `event_time`, `lookback`, and `batch_size` configurations to generate necessary filters for you, making the process more streamlined and reducing the need for you to manage these details.
* Handling updates: Use the `lookback` configuration to keep track of late-arriving records instead of you making that calculation.


runleonarun marked this conversation as resolved.
Show resolved Hide resolved
runleonarun marked this conversation as resolved.
Show resolved Hide resolved
### Snapshots improvements

Originally, snapshots were defined directly in the `dbt_project.yml` file, which involved YAML configuration for source and target schemas without any SQL logic. This method was cumbersome, as it limited flexibility and made managing snapshots more complex. Over time, snapshots evolved to use Jinja blocks, allowing for SQL logic within `.sql` files, but this added parsing complexity and made the development process less efficient.

Beginning in dbt Core 1.9, we've streamlined snapshot configuration by defining snapshots purely in YAML without any SQL logic. This improvement includes:

* New snapshot specification: Snapshots are now configured in a YAML file doe a cleaner more structured set up.
* New `snapshot_meta_column_names` config: Allows you to customize the names of meta fields (for example, `dbt_valid_from`, `dbt_valid_to`, etc.) that dbt automatically adds to snapshots. This increases flexibility to tailor metadata to your needs.
* `target_schema` now optional for snapshots: This schema is now optional When ommitted, snapshots will use the schema defined for the current environment.
* Standard schema and database configs supported: Snapshots will now be consistent with other dbt resources You can specify where snapshots should be stored.
* Warning for incorrect `updated_at` data type: To ensure data integrity, you'll see a warning if the `updated_at` field specified in the snapshot configuration is not the proper data type or timestamp.

### `state:modified` improvements

INSERT HERE Point me to a resource for this?
runleonarun marked this conversation as resolved.
Show resolved Hide resolved

Fewer false positives in state:modified
state_modified_compare_more_unrendered_values
state_modified_compare_vars

### Deprecated functionality

INSERT HERE - any deprecated functionality to call out?
runleonarun marked this conversation as resolved.
Show resolved Hide resolved


### Managing changes to legacy behaviors

dbt Core v1.9 has introduced flags for [managing changes to legacy behaviors](/reference/global-configs/behavior-changes). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `True` / `False` values, respectively, for `flags` in `dbt_project.yml`.

You can read more about each of these behavior changes in the following links:

INSERT HERE! Any behavior changes?
runleonarun marked this conversation as resolved.
Show resolved Hide resolved

## Quick hits

We also made some quality-of-life improvements in Core 1.9, enabling you to:

- Document [singular data tests](/docs/build/data-tests#document-singular-tests).
- Use `ref` and `source` in foreign key constraints
runleonarun marked this conversation as resolved.
Show resolved Hide resolved
- New CLI flag for `dbt test`. Choose which resource types are included or excluded when you run the `dbt test` by including [`--resource-type`/`--exclude-resource-type`](/reference/global-configs/resource-type)
runleonarun marked this conversation as resolved.
Show resolved Hide resolved
- New CLI flag for [`dbt show`](/reference/commands/show). `--inline-direct` enables you to avoid loading the entire manifest and
skip rendering any Jinja templates.
runleonarun marked this conversation as resolved.
Show resolved Hide resolved

We also made improvements for adapters, enabling you to:
- Use arbitrary config options in `data_test` For example, you can set `snowflake_warehouse` for tests.
- Use behavior flags INSERT HERE MORE INFO
Loading