Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-2016] [Spike] Static artifact for CLI validation #6840

Open
jtcohen6 opened this issue Feb 2, 2023 · 4 comments
Open

[CT-2016] [Spike] Static artifact for CLI validation #6840

jtcohen6 opened this issue Feb 2, 2023 · 4 comments

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Feb 2, 2023

The ask: A language-agnostic data structure to validate dbt CLI commands, without actually requiring dbt-core to be imported/installed. (Could be a JSONSchema, doesn't have to be.) Let's get away from any need for naïve regex.

For starters, the goal here wouldn't even be to parse CLI strings into meaningful representations — just to say, this is or isn't a valid CLI string. But I'd also imagine wanting to extend this nicer-to-have territory (auto-complete, blocking certain options, extending with additional options).

As I see it, two options:

  1. Abstract the combination of commands + params one step further than we already have, by means of Python methods & decorators, into a static data structure (e.g. JSON). Then, within dbt-core's CLI, consume that data structure to generate the CLI methods/decorators.
  2. Serialize click.cli to a static data structure, which could then be used (itself? by click? by another tool?) just to validate CLI strings.

The closest thing I could find built into click is the to_info_dict method, which is really intended to support auto-generating documentation:
https://click.palletsprojects.com/en/8.1.x/api/#click.Command.to_info_dict

>>> from click import Context
>>> from dbt.cli.main import cli
>>> with Context(cli) as ctx:
>>>   info = ctx.to_info_dict()
>>> info['command']['commands'].keys()
dict_keys(['build', 'clean', 'compile', 'debug', 'deps', 'docs', 'init', 'list', 'ls', 'parse', 'run', 'run-operation', 'seed', 'snapshot', 'source', 'test'])
>>> [param['name'] for param in info['command']['commands']['run']['params']]
['defer', 'favor_state', 'exclude', 'fail_fast', 'full_refresh', 'models', 'profile', 'profiles_dir', 'project_dir', 'select', 'selector', 'state', 'target', 'target_path', 'threads', 'vars', 'version_check', 'help']
@github-actions github-actions bot changed the title [Spike] Static artifact for CLI validation [CT-2016] [Spike] Static artifact for CLI validation Feb 2, 2023
@jtcohen6
Copy link
Contributor Author

jtcohen6 commented Feb 3, 2023

FYI to Execution team: I'm going to queue this up for estimation discussion. Not expecting a point estimate (since it's a spike), just expecting that you all know more than I do about this topic & might have some strong opinions!

@dbeatty10
Copy link
Contributor

dbeatty10 commented Jul 21, 2023

Here's a Python script that will output an artifact named dbt-core-cli-flags.json:

generate_cli_flags_artifact.py

import json
from click import Context
from dbt.cli.main import cli


def convert_to_serializable(obj):
    # Convert non-serializable objects to strings
    return str(obj)


def serialize_dict_to_json(input_dict):
    # Use a custom conversion function when serializing
    return json.dumps(input_dict, indent=4, default=convert_to_serializable)


with Context(cli) as ctx:
    info = ctx.to_info_dict()

    pretty_json_string = serialize_dict_to_json(info)

    # Write the JSON string to a file
    with open("dbt-core-cli-flags.json", "w") as file:
        file.write(pretty_json_string)

Usage

python generate_cli_flags_artifact.py

Note: the dictionary can contain content that isn't serializable to JSON (like tuples, functions, etc), so this script just converts those to a string.

Related internal Slack threads

(Added 2024-01-12)

@dbeatty10
Copy link
Contributor

Here is a script that produces the Markdown table below:

image

@dbeatty10
Copy link
Contributor

Here's a section of the documentation that outlines which sub-commands are available in dbt Core, dbt Cloud CLI, and dbt Cloud IDE:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants