[Bug]: FlinkRunner emits new output from Impulse after restoring from savepoint #1717
GitHub Actions / Test Results
failed
Apr 15, 2024 in 0s
1 fail, 16 skipped, 2 pass in 30m 24s
19 tests 2 ✅ 30m 24s ⏱️
1 suites 16 💤
1 files 1 ❌
Results for commit f071e9e.
Annotations
Check warning on line 0 in apache_beam.examples.wordcount_it_test.WordCountIT
github-actions / Test Results
test_wordcount_it_with_prebuilt_sdk_container_local_docker (apache_beam.examples.wordcount_it_test.WordCountIT) failed
sdks/python/pytest-beam_python3.10_sdk.xml [took 14m 5s]
Raw output
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Workflow failed. Causes: Job appears to be stuck. Several workers have failed to start up in a row, and no worker has successfully started up for this job. Last error reported: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/github-actions/prebuild_python310_sdk/beam_python_prebuilt_sdk:0b226047-bfd1-4522-90ed-d9f0de8a7966 not found: manifest unknown: Failed to fetch "0b226047-bfd1-4522-90ed-d9f0de8a7966" from request "/v2/apache-beam-testing/github-actions/prebuild_python310_sdk/beam_python_prebuilt_sdk/manifests/0b226047-bfd1-4522-90ed-d9f0de8a7966".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image..
self = <apache_beam.examples.wordcount_it_test.WordCountIT testMethod=test_wordcount_it_with_prebuilt_sdk_container_local_docker>
@pytest.mark.it_validatescontainer
def test_wordcount_it_with_prebuilt_sdk_container_local_docker(self):
> self._run_wordcount_it(
wordcount.run,
experiment='beam_fn_api',
prebuild_sdk_container_engine='local_docker')
apache_beam/examples/wordcount_it_test.py:95:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/examples/wordcount_it_test.py:150: in _run_wordcount_it
run_wordcount(
apache_beam/examples/wordcount.py:87: in run
with beam.Pipeline(options=pipeline_options) as p:
apache_beam/pipeline.py:613: in __exit__
self.result = self.run()
apache_beam/pipeline.py:563: in run
self._options).run(False)
apache_beam/pipeline.py:587: in run
return self.runner.run_pipeline(self, self._options)
apache_beam/runners/dataflow/test_dataflow_runner.py:66: in run_pipeline
self.result.wait_until_finish(duration=wait_duration)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <DataflowPipelineResult <Job
clientRequestId: '20240415115516533663-4621'
createTime: '2024-04-15T11:55:17.538312Z'
...024-04-15T11:55:17.538312Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> at 0x78cfa6a90d60>
duration = None
def wait_until_finish(self, duration=None):
if not self.is_in_terminal_state():
if not self.has_job:
raise IOError('Failed to get the Dataflow job id.')
consoleUrl = (
"Console URL: https://console.cloud.google.com/"
f"dataflow/jobs/<RegionId>/{self.job_id()}"
"?project=<ProjectId>")
thread = threading.Thread(
target=DataflowRunner.poll_for_job_completion,
args=(self._runner, self, duration))
# Mark the thread as a daemon thread so a keyboard interrupt on the main
# thread will terminate everything. This is also the reason we will not
# use thread.join() to wait for the polling thread.
thread.daemon = True
thread.start()
while thread.is_alive():
time.sleep(5.0)
# TODO: Merge the termination code in poll_for_job_completion and
# is_in_terminal_state.
terminated = self.is_in_terminal_state()
assert duration or terminated, (
'Job did not reach to a terminal state after waiting indefinitely. '
'{}'.format(consoleUrl))
if terminated and self.state != PipelineState.DONE:
# TODO(BEAM-1290): Consider converting this to an error log based on
# theresolution of the issue.
_LOGGER.error(consoleUrl)
> raise DataflowRuntimeException(
'Dataflow pipeline failed. State: %s, Error:\n%s' %
(self.state, getattr(self._runner, 'last_error_msg', None)),
E apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
E Workflow failed. Causes: Job appears to be stuck. Several workers have failed to start up in a row, and no worker has successfully started up for this job. Last error reported: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/github-actions/prebuild_python310_sdk/beam_python_prebuilt_sdk:0b226047-bfd1-4522-90ed-d9f0de8a7966 not found: manifest unknown: Failed to fetch "0b226047-bfd1-4522-90ed-d9f0de8a7966" from request "/v2/apache-beam-testing/github-actions/prebuild_python310_sdk/beam_python_prebuilt_sdk/manifests/0b226047-bfd1-4522-90ed-d9f0de8a7966".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image..
apache_beam/runners/dataflow/dataflow_runner.py:780: DataflowRuntimeException
Check notice on line 0 in .github
github-actions / Test Results
16 skipped tests found
There are 16 skipped tests, see "Raw output" for the full list of skipped tests.
Raw output
apache_beam.examples.ml_transform.ml_transform_it_test
apache_beam.examples.snippets.transforms.elementwise.mltransform_test
apache_beam.examples.snippets.transforms.elementwise.runinference_test
apache_beam.ml.inference.huggingface_inference_it_test
apache_beam.ml.inference.huggingface_inference_test
apache_beam.ml.inference.onnx_inference_test
apache_beam.ml.inference.pytorch_inference_test
apache_beam.ml.inference.tensorrt_inference_test
apache_beam.ml.inference.vertex_ai_inference_it_test
apache_beam.ml.inference.xgboost_inference_test
apache_beam.ml.transforms.handlers_test
apache_beam.ml.transforms.tft_test
apache_beam.runners.dask.dask_runner_test
apache_beam.testing.analyzers.perf_analysis_test
apache_beam.testing.benchmarks.cloudml.cloudml_benchmark_test
apache_beam.typehints.pytorch_type_compatibility_test
Check notice on line 0 in .github
github-actions / Test Results
19 tests found
There are 19 tests, see "Raw output" for the full list of tests.
Raw output
apache_beam.examples.ml_transform.ml_transform_it_test
apache_beam.examples.snippets.transforms.elementwise.mltransform_test
apache_beam.examples.snippets.transforms.elementwise.runinference_test
apache_beam.examples.wordcount_it_test.WordCountIT ‑ test_wordcount_it
apache_beam.examples.wordcount_it_test.WordCountIT ‑ test_wordcount_it_with_prebuilt_sdk_container_cloud_build
apache_beam.examples.wordcount_it_test.WordCountIT ‑ test_wordcount_it_with_prebuilt_sdk_container_local_docker
apache_beam.ml.inference.huggingface_inference_it_test
apache_beam.ml.inference.huggingface_inference_test
apache_beam.ml.inference.onnx_inference_test
apache_beam.ml.inference.pytorch_inference_test
apache_beam.ml.inference.tensorrt_inference_test
apache_beam.ml.inference.vertex_ai_inference_it_test
apache_beam.ml.inference.xgboost_inference_test
apache_beam.ml.transforms.handlers_test
apache_beam.ml.transforms.tft_test
apache_beam.runners.dask.dask_runner_test
apache_beam.testing.analyzers.perf_analysis_test
apache_beam.testing.benchmarks.cloudml.cloudml_benchmark_test
apache_beam.typehints.pytorch_type_compatibility_test
Loading