[Bug] dbt list --output json erroring with Object of type date is not JSON serializable
#10556
Comments
|
@jeremyyeo Summarizing your findings, is my understanding correct?
We think we have a fix for 1.8, can you confirm if this issue occurs in 1.7 + core? Thanks! |
|
I think I'd put it this way: On dbt-core 1.6, 1.7, 1.8 using dbt-core CLI - this errors whenever we explicitly issue a command to output json In dbt Cloud - on 1.7:
^ There is no initial error.
^ Up until you try to do what we did in core CLI - invoke a command and explicitly set In dbt Cloud - on versionless (1.8+): It immediately errored (screenshot copied from above when I opened the issue - see note below): 
Which is blocking any further operation - users cant run anything. There's something special about versionless in Cloud attempting to do json things behind the scenes - that isn't done on 1.7. Note: As of this writing though - this is no longer the case on versionless:
Likely cause of the reversion of the streaming manifest internal PR. Essentially:
|
|
Here's the same reproducible example supplied by @jeremyyeo originally, but simplified slightly: Reprex
{{ config(
start_date=modules.datetime.date(2024, 5, 1),
)
}}
select '{{ config.get("start_date") }}' as cdbt list --output jsonWorkaroundBut if the start_date=modules.datetime.date(2024, 5, 1) | string,dbt list --output json{"name": "foo", "resource_type": "model", "package_name": "my_project", "original_file_path": "models/foo.sql", "unique_id": "model.my_project.foo", "alias": "foo", "config": {"enabled": true, "alias": null, "schema": null, "database": null, "tags": [], "meta": {}, "group": null, "materialized": "view", "incremental_strategy": null, "persist_docs": {}, "post-hook": [], "pre-hook": [], "quoting": {}, "column_types": {}, "full_refresh": null, "unique_key": null, "on_schema_change": "ignore", "on_configuration_change": "apply", "grants": {}, "packages": [], "docs": {"show": true, "node_color": null}, "contract": {"enforced": false, "alias_types": true}, "access": "protected", "start_date": "2024-05-01"}, "tags": [], "depends_on": {"macros": [], "nodes": []}} |
|
This is setting a custom field in the node config, correct? So it just goes into the _extra dictionary, and since there is no type associated with those dictionary keys, it's not serializable. I think the possibilities here are 1) cast everything that's set in an extra key to string, or 2) raise an error when somebody sets an extra key to anything except a string. |
|
Note that the same is true of the "meta" dictionary. |
|
@gshank What are we doing right now with the "meta" dictionary? Does it have the same issue as reported here? Or are we 1) casting everything to a string or 2) raising an error for non-strings? |
|
We don't do anything different for meta, just like extra keys. I think that people might be less likely to set it in the sql file though? Because it would be difficult or impossible to get objects set via yaml, so this is mostly an issue from the greater flexibility that people have in setting the keys in jinja. |
|
@gshank Per your options in #10556 (comment), I'd advocate for:
That way there isn't any surprising gotchas due to us doing an implicit type cast to string. Rather, we'd ask the user to do an explicit type cast of their choosing. We could do an error message like this:
|
If we do this - hopefully it not an exception and potentially breaks users existing runs. Can already predict having to explain to existing customers who have this pattern on versionless that we accidentally broke their runs - and then them asking us what about our promise of versionless not breaking runs. Just thought I'd call this out while ya'll figure out the best path forward :) |
💯 @jeremyyeo -- thanks for calling this out. Maybe we could just catch errors like What do you think @gshank? |
|
Actually, to reproduce this error it's not necessary to input Python-like code like Assuming I have a model {% if execute %}
{% for var_name in ['SOME_VAR', 'ANOTHER_VAR'] %}
{{ print(var_name ~ " value: " ~ var(var_name)) }}
{{ print(var_name ~ " repr: " ~ "{0!r}".format(var(var_name))) }}
{% endfor %}
{% endif %}This model just prints out two required variable values ( Now I use dbt Core As you can see, both variables have been converted into
Now Finally, we can reproduce the issue - we just need to forget to pass a value for the required variable ( As you can see, I never indicated that I was passing a date. The YAML parser itself understood that the string Now we see the expected error message (about missing required var |
|
This error was driving me crazy, but @StanleySane 's workaround of forcing YAML string tag in |
|
I'm experiencing this error using and in The error appears when running Casting the variable to string with jinja templating does not work however. |
|
@internetcoffeephone Did you find any solution for this ? I am facing this problem too |
Is this a new bug in dbt-core?
Current Behavior
Doing a list with json output results in an error
Object of type date is not JSON serializable.Expected Behavior
No error?
Steps To Reproduce
Relevant log output
No response
Environment
Which database adapter are you using with dbt?
postgres, snowflake
Additional Context
Originally raised: https://getdbt.slack.com/archives/CBSQTAPLG/p1722637389979709
There's an interesting thing happening with dbt Cloud though... when on 1.7, we don't emit an error
But when users go to versionless:
We can see from repro above that on 1.7, we can also trigger this error if we do a json output... so it's not specifically a 1.8 thing - but rather a 1.8/versionless thing when in dbt Cloud.
We can trigger this error on 1.7 in cloud like so:
Couple of similar oldies:
#2373
#5357
The text was updated successfully, but these errors were encountered: