Have a page that outlines available flags and subcommands

[mirnawong1]

Contributions

• I have read the contribution docs, and understand what's expected of me.

What page(s) or areas on docs.getdbt.com are affected?

We don’t have a clear docs page/section outlining the available flags and which dbt command they can use.

this seems to be a pretty important gap not covered in the docs.

• One of the suggestions was having a table listing all those options and highlighting dbt commands (and sub commands) compatibility. (and possibly also what flags are available in the ide and cli).
• another suggestion was whether we can automatically have a pr or inform the docs team whenever there’s a change/addition to the dbt --help flag.

What changes are you suggesting?

Having a table that lists all of the flags and sub commands, what commands those flags are compatible with, and which editing tool supports those flags.

Additional information

https://docs.getdbt.com/reference/global-configs/failing-fast

[dbeatty10]

This would be easiest to maintain if there were some process that autogenerates the page(s) with the flags + sub commands.

Here is a script that uses click to determine which CLI flags (and environment variables) are relevant to each sub-command.

[dbeatty10]

@aranke and I discussed this issue today while exploring options for dbt-labs/dbt-core#9575.

Example

Details

Here's a pairing of scripts that produces that markdown table:

python generate_cli_flags_artifact.py
python generate_markdown.py > pivot_table.md

generate_cli_flags_artifact.py is from here.

generate_markdown.py

import pandas as pd
import json


def list_to_pd(result_list):
    return pd.DataFrame(result_list)


def extract_commands_and_params(filename):
    with open(filename, "r") as file:
        data = json.load(file)
    
    commands_list = []
    parent_command = "dbt"

    if data.get("command") and data["command"].get("commands"):
        add_subcommands(data["command"]["commands"], parent_command, commands_list)

    return commands_list


def add_subcommands(commands, parent_command="dbt", commands_list=[]):

    # base case
    if not commands:
        return

    for command_name, command_info in commands.items():

        full_command_name = f"{parent_command} {command_name}"

        if command_info["params"]:
            for param in command_info["params"]:
                flag_name = f"{param['name']}"
                commands_list.append({"subcommand": full_command_name, "option": flag_name})
        else:
            # If there are no params, still add the command with an empty option
            commands_list.append({"subcommand": full_command_name, "option": ""})

        # recursively explore any sub-commands
        if command_info.get("commands"):
            add_subcommands(command_info["commands"], full_command_name, commands_list)


def main():

    # Assuming the JSON data is stored in a file named dbt-core-cli-flags.json
    filename = "dbt-core-cli-flags.json"
    click_options = extract_commands_and_params(filename)
    result_df = list_to_pd(click_options)

    # Represents the presence of a pair with the value 1
    result_df["count"] = 1

    # Create a pivot table
    table = pd.pivot_table(
        result_df,
        values="count",
        index=["option"],
        columns=["subcommand"],
        aggfunc="sum",
        fill_value=0,
    )

    # Convert 1's and 0's to applicable icons
    table.replace(to_replace=1, value="✅", inplace=True)
    table.replace(to_replace=0, value="", inplace=True)

    # Export to Markdown
    print(table.to_markdown(stralign="center"))


if __name__ == "__main__":
    main()

[dbeatty10]

#4814 resolved #2894 and added the big table of available flags (global configs).

But there are still non-global CLI flag / environment variable combos that aren't documented. One example is --empty / DBT_EMPTY which is only available for compile, run, and build.