Skip to content

Commit

Permalink
Add docs about blacklist and progenitor option (#91)
Browse files Browse the repository at this point in the history
* add docs for blacklist

* add docs about progenitor option
  • Loading branch information
syou6162 authored Sep 11, 2023
1 parent f284899 commit 015416c
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 0 deletions.
6 changes: 6 additions & 0 deletions docs/docs/tutorial-yaml/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,14 @@ vars:
path: <path>
database: <database>
schema: <schema>
_blacklist: <blacklist>
```

- `<source_name>` is the name of a source in your `dbt_project.yml` file.
- `<path>` is the path to the YAML file that will be generated for the source. This path is relative to the root of your dbt project models directory.
- `<database>` is the database that will be used for the source. If not specified, the database will default to the one in your profiles.yml file.
- `<schema>` is the schema that will be used for the source. If not specified, the source name is assumed to be the schema which matches dbt's default behavior.
- `<blacklist>` is the columns to be ignored. You can use regular expressions to specify which columns you'd like to exclude.

#### Examples

Expand All @@ -76,6 +78,10 @@ vars:
# a dedicated directory for all sources
github: "all_sources/github.yml"
_blacklist:
- "_FIVETRAN_SYNCED"
- ".*__key__.namespace"
```

Notice the use of the `{parent}` variable in the `jira` source configuration. This variable is a special variable that will be replaced with the name of the parent directory of the YAML file. The other special variables are `{node}` and `{model}`. We will discuss these variables in more detail in the next section.
6 changes: 6 additions & 0 deletions docs/docs/tutorial-yaml/inheritance.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ dbt-osmosis accumulates a knowledge graph for a specfic model by traversing the

The crux of the value proposition is that we often alias columns in our staging models and use them many times in many places without changing the name. This means, within the context of a specific models family tree, we should be able to inherit that knowledge. This inheritance can include tags and descriptions. This permits propagating PII, GDPR, and other compliance related tags for example. When a column is used in a model and its definition is semantically different while the column name is the same (which is a questionable practice), you should update the definition for that column in that model. The inheritors will use the updated definition if they pull from said model.

:::tip Tip

If you use the `--add-progenitor-to-meta` option, you can add the `meta.osmosis_progenitor` field to yaml files, indicating which model is inherited from.

:::

## Takeways

While this obviously has some limitations, it is a powerful feature that can help you be more DRY with your documentation. It is also a great way to ensure that your documentation is consistent across your entire dbt project. It captures +80% of the meat of the repetitive documentation for a model.
Expand Down

0 comments on commit 015416c

Please sign in to comment.