Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix to load dbt vars using yaml instead of json #107

Merged
merged 1 commit into from
Sep 22, 2023
Merged

Fix to load dbt vars using yaml instead of json #107

merged 1 commit into from
Sep 22, 2023

Conversation

yu-iskw
Copy link
Contributor

@yu-iskw yu-iskw commented Sep 21, 2023

Overview

When I make the pull request #99 fresh with git merge --ff, the implementation to load the values passed by the --vers option was degraded.

Test

Test without --vars

$ dbt-osmosis yaml refactor  --profiles-dir . --project-dir .
INFO     � Executing dbt-osmosis                                                                                                    main.py:197

INFO     � Searching project stucture for required updates and building action plan                                              osmosis.py:619
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Project structure approved                                                                                            osmosis.py:653
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Processing model: model.jaffle_shop.orders                                                                            osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.customers_alias                                                                   osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_customers                                                                     osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_payments                                                                      osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_orders                                                                        osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     ✨ Schema file is up to date for model model.jaffle_shop.orders                                                          osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_customers                                                              osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_customers                                                   osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_payments                                                               osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_payments                                                    osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_orders                                                                 osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_orders                                                      osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.customers_alias                                                            osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.customers_alias                                                 osmosis.py:906

Test with --vars of a JSON file

$ cat test-vars.json
{
  "a": "1",
  "b": {
    "b1": 1,
    "b2": [1, 2, 3],
    "b3": ["1", "2", "3"]
  }
}

$ dbt-osmosis yaml refactor --vars "$(cat test-vars.json)" --profiles-dir . --project-dir .
INFO     � Executing dbt-osmosis                                                                                                    main.py:197

INFO     � Searching project stucture for required updates and building action plan                                              osmosis.py:619
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Project structure approved                                                                                            osmosis.py:653
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Processing model: model.jaffle_shop.orders                                                                            osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.customers_alias                                                                   osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_customers                                                                     osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_payments                                                                      osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_orders                                                                        osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Looking for actions for model.jaffle_shop.stg_customers                                                              osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_customers                                                   osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_payments                                                               osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_payments                                                    osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.customers_alias                                                            osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.customers_alias                                                 osmosis.py:906
INFO     ✨ Schema file is up to date for model model.jaffle_shop.orders                                                          osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_orders                                                                 osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_orders                                                      osmosis.py:906

Test with --vars of a YAML file

$ cat test-vars.yml
---
a: '1'
b:
  b1: 1
  b2:
  - 1
  - 2
  - 3
  b3:
  - '1'
  - '2'
  - '3'

$ dbt-osmosis yaml refactor --vars "$(cat test-vars.yml)" --profiles-dir . --project-dir .
INFO     � Executing dbt-osmosis                                                                                                    main.py:197

INFO     � Searching project stucture for required updates and building action plan                                              osmosis.py:619
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Project structure approved                                                                                            osmosis.py:653
INFO     ...building project structure mapping in memory                                                                          osmosis.py:500
INFO     � Processing model: model.jaffle_shop.orders                                                                            osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.customers_alias                                                                   osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_customers                                                                     osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_payments                                                                      osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Processing model: model.jaffle_shop.stg_orders                                                                        osmosis.py:792
INFO     � Resolving columns in database                                                                                         osmosis.py:803
INFO     � Looking for actions for model.jaffle_shop.customers_alias                                                            osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.customers_alias                                                 osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_customers                                                              osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_customers                                                   osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_orders                                                                 osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_orders                                                      osmosis.py:906
INFO     ✨ Schema file is up to date for model model.jaffle_shop.orders                                                          osmosis.py:906
INFO     � Looking for actions for model.jaffle_shop.stg_payments                                                               osmosis.py:1105
INFO     ✨ Schema file is up to date for model model.jaffle_shop.stg_payments                                                    osmosis.py:906

@yu-iskw yu-iskw marked this pull request as ready for review September 21, 2023 06:44
@yu-iskw
Copy link
Contributor Author

yu-iskw commented Sep 21, 2023

@z3z1ma When I resolve the conflicts in #99, the implementation to load the value of --vars was degraded. So, I fixed the issue. Can you review this?

@syou6162 syou6162 mentioned this pull request Sep 21, 2023
Closed
@z3z1ma
Copy link
Owner

z3z1ma commented Sep 22, 2023

lgtm

@z3z1ma z3z1ma merged commit fca3033 into z3z1ma:main Sep 22, 2023
@yu-iskw
Copy link
Contributor Author

yu-iskw commented Sep 22, 2023

@z3z1ma I look forward to the next release. Thank you for your understanding and cooperation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yu-iskw @z3z1ma