[Feature] - User Defined Macros & Filters

[afranzi]

Contributions

• I have read the contribution docs, and understand what's expected of me.

Link to the page on docs.getdbt.com requiring updates

Not sure if this would be part of the https://docs.getdbt.com/reference/project-configs/macro-paths
Or in https://docs.getdbt.com/reference/dbt-jinja-functions

What part(s) of the page would you like to see updated?

The main idea would be adding documentation about the new incoming feature allowing DBT developers to inject their own python JINJA macros and filters in addition to the native ones, so developers can move complex jinja templating code to python and level up their dbt projects with more possibilities.

Some examples:

• Expand DBT credentials fetching them from a secret store (i.e. AWS Secrets Manager).
• Enable models based on an external config (i.e. HTTP Requests to an external API which defines if the model can be enabled).
  by

We can enable this behavior by defining the following ENV vars with the path:instance format.

• DBT_USER_DEFINED_MACROS: Jinja Macros
• DBT_USER_DEFINED_FILTERS: Jinja Filters

Additional information

This documentation would be based on this PR dbt-labs/dbt-common#72.

See following example about how I've used it locally:

[afranzi]

Adding some examples about the credentials:

[ismailsimsek]

created related discussion here dbt-labs/dbt-adapters#259

IMO Adapter factory should allow user provided customized adapter class. this will open door to customization

[ismailsimsek]

PR dbt-labs/dbt-adapters#261
implemented in this package https://github.com/memiiso/opendbt
example https://github.com/memiiso/opendbt/blob/main/docs/EXAMPLES.md#execute-python-model-locally

using @available decorator makes the function available to jinja

[matthewshaver]

Attached PRs were closed, so closing this issue as well.

[afranzi]

It's quite sad seeing DBT Labs taking this approach and closing it.