Skip to content
This repository has been archived by the owner on Jul 24, 2024. It is now read-only.

Add missing dbt commands #68

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

dinigo
Copy link

@dinigo dinigo commented Sep 5, 2022

According to the DBT CLI reference there are still some missing commands.

This solves:

mock_dag,
):
"""Every operator passess down to the execution the correct dbt command"""
task_id = 'test_dbt_' + '_'.join(expected_command)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flake8 complains about an extra space here

class DbtRunOperationOperator(DbtBaseOperator):
@apply_defaults
def __init__(self, profiles_dir=None, target=None, *args, **kwargs):
super(DbtRunOperationOperator, self).__init__(profiles_dir=profiles_dir, target=target, *args, **kwargs)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flake8 complains about too long line here

@Gwildor
Copy link

Gwildor commented May 16, 2023

I think the DbtRunOperationOperator requires a bit more work. It is, as far as I know, the only dbt command that takes a positional argument (the macro you want to run), and uses --args (why they don't use --vars like other commands I don't know) for passing extra flags to the operation (it's the only command I could find with that argument).

I got it working by doing something like this:

class DbtBaseOperator:

    @apply_defaults
    def __init__(self,
                 macro=None,
                 args=None,
                 *positional_args,
                 **kwargs):
        super(DbtBaseOperator, self).__init__(*positional_args, **kwargs)

        if args is not None and not isinstance(args, dict):
            raise AirflowException('The argument "args" should be a dictionary')

        self.macro = macro
        self.args = args

    def create_hook(self):
        self.hook = DbtHook(
            macro=self.macro,
            args=self.args)

        return self.hook

class DbtRunOperationOperator(DbtBaseOperator):
    @apply_defaults
    def __init__(self, profiles_dir=None, target=None, *args, **kwargs):
        super(DbtRunOperationOperator, self).__init__(profiles_dir=profiles_dir, target=target, *args, **kwargs)

    def execute(self, context):
        self.create_hook().execute('run-operation', self.macro)

And then of course adding the --args keyword in the hook as well. I don't run dbt locally, so I don't have an example patch for that.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add DbtSourceFreshnessOperator Add support for dbt-build-operator
4 participants