Skip to content

[Bug]: FlinkRunner emits new output from Impulse after restoring from savepoint #1717

[Bug]: FlinkRunner emits new output from Impulse after restoring from savepoint

[Bug]: FlinkRunner emits new output from Impulse after restoring from savepoint #1717

GitHub Actions / Test Results failed Apr 15, 2024 in 0s

1 fail, 16 skipped, 2 pass in 30m 53s

19 tests    2 ✅  30m 53s ⏱️
 1 suites  16 💤
 1 files     1 ❌

Results for commit f071e9e.

Annotations

Check warning on line 0 in apache_beam.examples.wordcount_it_test.WordCountIT

See this annotation in the file changed.

@github-actions github-actions / Test Results

test_wordcount_it_with_prebuilt_sdk_container_local_docker (apache_beam.examples.wordcount_it_test.WordCountIT) failed

sdks/python/pytest-beam_python3.9_sdk.xml [took 14m 29s]
Raw output
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Workflow failed. Causes: Job appears to be stuck. Several workers have failed to start up in a row, and no worker has successfully started up for this job. Last error reported: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/github-actions/prebuild_python39_sdk/beam_python_prebuilt_sdk:dd91f6f6-693a-40f3-9247-f251eaf43b41 not found: manifest unknown: Failed to fetch "dd91f6f6-693a-40f3-9247-f251eaf43b41" from request "/v2/apache-beam-testing/github-actions/prebuild_python39_sdk/beam_python_prebuilt_sdk/manifests/dd91f6f6-693a-40f3-9247-f251eaf43b41".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image..
self = <apache_beam.examples.wordcount_it_test.WordCountIT testMethod=test_wordcount_it_with_prebuilt_sdk_container_local_docker>

    @pytest.mark.it_validatescontainer
    def test_wordcount_it_with_prebuilt_sdk_container_local_docker(self):
>     self._run_wordcount_it(
          wordcount.run,
          experiment='beam_fn_api',
          prebuild_sdk_container_engine='local_docker')

apache_beam/examples/wordcount_it_test.py:95: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apache_beam/examples/wordcount_it_test.py:150: in _run_wordcount_it
    run_wordcount(
apache_beam/examples/wordcount.py:106: in run
    output | 'Write' >> WriteToText(known_args.output)
apache_beam/pipeline.py:613: in __exit__
    self.result = self.run()
apache_beam/pipeline.py:560: in run
    return Pipeline.from_runner_api(
apache_beam/pipeline.py:587: in run
    return self.runner.run_pipeline(self, self._options)
apache_beam/runners/dataflow/test_dataflow_runner.py:66: in run_pipeline
    self.result.wait_until_finish(duration=wait_duration)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <DataflowPipelineResult <Job
 clientRequestId: '20240415115450107867-4621'
 createTime: '2024-04-15T11:54:51.198705Z'
...024-04-15T11:54:51.198705Z'
 steps: []
 tempFiles: []
 type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> at 0x7da79cbae460>
duration = None

    def wait_until_finish(self, duration=None):
      if not self.is_in_terminal_state():
        if not self.has_job:
          raise IOError('Failed to get the Dataflow job id.')
        consoleUrl = (
            "Console URL: https://console.cloud.google.com/"
            f"dataflow/jobs/<RegionId>/{self.job_id()}"
            "?project=<ProjectId>")
        thread = threading.Thread(
            target=DataflowRunner.poll_for_job_completion,
            args=(self._runner, self, duration))
    
        # Mark the thread as a daemon thread so a keyboard interrupt on the main
        # thread will terminate everything. This is also the reason we will not
        # use thread.join() to wait for the polling thread.
        thread.daemon = True
        thread.start()
        while thread.is_alive():
          time.sleep(5.0)
    
        # TODO: Merge the termination code in poll_for_job_completion and
        # is_in_terminal_state.
        terminated = self.is_in_terminal_state()
        assert duration or terminated, (
            'Job did not reach to a terminal state after waiting indefinitely. '
            '{}'.format(consoleUrl))
    
        if terminated and self.state != PipelineState.DONE:
          # TODO(BEAM-1290): Consider converting this to an error log based on
          # theresolution of the issue.
          _LOGGER.error(consoleUrl)
>         raise DataflowRuntimeException(
              'Dataflow pipeline failed. State: %s, Error:\n%s' %
              (self.state, getattr(self._runner, 'last_error_msg', None)),
E             apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
E             Workflow failed. Causes: Job appears to be stuck. Several workers have failed to start up in a row, and no worker has successfully started up for this job. Last error reported: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/github-actions/prebuild_python39_sdk/beam_python_prebuilt_sdk:dd91f6f6-693a-40f3-9247-f251eaf43b41 not found: manifest unknown: Failed to fetch "dd91f6f6-693a-40f3-9247-f251eaf43b41" from request "/v2/apache-beam-testing/github-actions/prebuild_python39_sdk/beam_python_prebuilt_sdk/manifests/dd91f6f6-693a-40f3-9247-f251eaf43b41".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image..

apache_beam/runners/dataflow/dataflow_runner.py:780: DataflowRuntimeException

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Test Results

16 skipped tests found

There are 16 skipped tests, see "Raw output" for the full list of skipped tests.
Raw output
apache_beam.examples.ml_transform.ml_transform_it_test
apache_beam.examples.snippets.transforms.elementwise.mltransform_test
apache_beam.examples.snippets.transforms.elementwise.runinference_test
apache_beam.ml.inference.huggingface_inference_it_test
apache_beam.ml.inference.huggingface_inference_test
apache_beam.ml.inference.onnx_inference_test
apache_beam.ml.inference.pytorch_inference_test
apache_beam.ml.inference.tensorrt_inference_test
apache_beam.ml.inference.vertex_ai_inference_it_test
apache_beam.ml.inference.xgboost_inference_test
apache_beam.ml.transforms.handlers_test
apache_beam.ml.transforms.tft_test
apache_beam.runners.dask.dask_runner_test
apache_beam.testing.analyzers.perf_analysis_test
apache_beam.testing.benchmarks.cloudml.cloudml_benchmark_test
apache_beam.typehints.pytorch_type_compatibility_test

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Test Results

19 tests found

There are 19 tests, see "Raw output" for the full list of tests.
Raw output
apache_beam.examples.ml_transform.ml_transform_it_test
apache_beam.examples.snippets.transforms.elementwise.mltransform_test
apache_beam.examples.snippets.transforms.elementwise.runinference_test
apache_beam.examples.wordcount_it_test.WordCountIT ‑ test_wordcount_it
apache_beam.examples.wordcount_it_test.WordCountIT ‑ test_wordcount_it_with_prebuilt_sdk_container_cloud_build
apache_beam.examples.wordcount_it_test.WordCountIT ‑ test_wordcount_it_with_prebuilt_sdk_container_local_docker
apache_beam.ml.inference.huggingface_inference_it_test
apache_beam.ml.inference.huggingface_inference_test
apache_beam.ml.inference.onnx_inference_test
apache_beam.ml.inference.pytorch_inference_test
apache_beam.ml.inference.tensorrt_inference_test
apache_beam.ml.inference.vertex_ai_inference_it_test
apache_beam.ml.inference.xgboost_inference_test
apache_beam.ml.transforms.handlers_test
apache_beam.ml.transforms.tft_test
apache_beam.runners.dask.dask_runner_test
apache_beam.testing.analyzers.perf_analysis_test
apache_beam.testing.benchmarks.cloudml.cloudml_benchmark_test
apache_beam.typehints.pytorch_type_compatibility_test