Skip to content

Commit

Permalink
Do not block Cosmos if openlineage-common raises a jinja exception (#626
Browse files Browse the repository at this point in the history
)

Before this change, Cosmos failed to run if there is an issue in the
openlineage-common parsing of the dbt project due to a jinja2 exception,
which did not happen when running the dbt command by itself:

#612 (comment)

```
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 268, in calculate_openlineage_events_completes
    openlineage_processor = DbtLocalArtifactProcessor(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 57, in __init__
    dbt_project = self.load_yaml_with_jinja(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 157, in load_yaml_with_jinja
    return self.render_values_jinja(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja
    parsed_dict[key] = cls.render_values_jinja(
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja
    parsed_dict[key] = cls.render_values_jinja(
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja
    parsed_dict[key] = cls.render_values_jinja(
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 190, in render_values_jinja
    return environment.from_string(value).render()  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: 'run_started_at' is undefined
```
Closes: #612
Relates to: OpenLineage/OpenLineage#2212

(cherry picked from commit 4c65f25)
  • Loading branch information
tatiana committed Oct 25, 2023
1 parent a36bdc4 commit 291dc62
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion cosmos/operators/local.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from typing import Any, Callable, Literal, Sequence, TYPE_CHECKING

import airflow
import jinja2
import yaml
from airflow import DAG
from airflow.compat.functools import cached_property
Expand Down Expand Up @@ -281,7 +282,7 @@ def calculate_openlineage_events_completes(
try:
events = openlineage_processor.parse()
self.openlineage_events_completes = events.completes
except (FileNotFoundError, NotImplementedError, ValueError, KeyError):
except (FileNotFoundError, NotImplementedError, ValueError, KeyError, jinja2.exceptions.UndefinedError):
logger.debug("Unable to parse OpenLineage events", stack_info=True)

def get_datasets(self, source: Literal["inputs", "outputs"]) -> list[Dataset]:
Expand Down

0 comments on commit 291dc62

Please sign in to comment.