failure_hook not called in testing

[ahoneycutt]

Hello, I'm struggling to get my failure hook to be called when a solid constructed using with_hooks raises an exception. The failure_hook is supposed to call a method on a resource (in this case the resource is mocked with hardcoded_resource), but the failure hook isn't called at all.

Below is an example (mostly pseudo-code) to help illustrate my problem.

"""my_pipeline.py"""
def pipeline():
    data = get_some_data_solid.with_hooks({my_failure_hook})()


@failure_hook(required_resource_keys={'my_resource'})
def my_failure_hook(context):
    context.resources.my_resource.do_failure()

"""test_my_pipeline.py"""
class PipelineTest(unittest.TestCase):
    def setUp(self) -> None:
        # self.fail_mock is injected into hardcoded_resource for
        # the resource `my_resource` which should cause a raised exception in 
        # solid
        #
        # EDIT: fail_mock mocks the `do_failure` method in `my_resource`, an additional mock is used
        #           on the same resource which will raise an exception in the solid
        self.fail_mock = mock.MagicMock()
        # Get mode definition, using a hardcoded resource mock
        # which will cause exception to be raised in solid
        # ...
        self.pipeline = @decorated_pipeline_with_mode_def

    def test_pipeline(self):
        with self.assertRaises(Exception):
            execute_pipeline(
                pipeline=self.pipeline,
                mode='test',
            )
        self.fail_mock.assert_called_once()

Does mocking the resource used in the failure hook somehow break the hook attachment in the pipeline? Perhaps I just have an error in my approach.

Thanks for any help here,
Andy

[ahoneycutt]

The answer here is to pass raise_on_error=False to the call to execute_pipeline, i.e. execute_pipeline(pipeline=self.pipeline, mode='test', raise_on_error=False) as the pipeline was halting when the solid raised an exception.

By default, as mentioned by the Dagster team, raise_on_error is True in the execute_pipeline command (and only available to toggle here), and the default behavior for pipeline execution is otherwise equivalent False everywhere else.