[Failing Test]: PostCommit Python and PostCommit Python Arm perma red

[damondouglas]

What happened?

PostCommit Python and PostCommit Python Arm in perma red status.

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[shunping]

I took a look at the logs in PostCommit Python.

• The last successful run was PostCommit Python #2719 @ 04/26/2024 1:32PM EDT.
• The first failure has occurred since PostCommit Python #2721 @ 04/26/2024 7:32PM EDT.
• Here is the traceback:

Traceback (most recent call last):
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 192, in console_main
    code = main()
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 150, in main
    config = _prepareconfig(args, plugins)
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 331, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_callers.py", line 156, in _multicall
    teardown[0].send(outcome)
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/helpconfig.py", line 104, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1075, in pytest_cmdline_parse
    self.parse(args)
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1425, in parse
    self._preparse(args, addopts=addopts)
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1305, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/pluggy/_manager.py", line 421, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 186, in exec_module
    exec(co, module.__dict__)
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/typeguard/__init__.py", line 21, in <module>
    from ._importhook import ImportHookManager as ImportHookManager
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 186, in exec_module
    exec(co, module.__dict__)
  File "/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/typeguard/_importhook.py", line 22, in <module>
    from typing_extensions import Buffer
ImportError: cannot import name 'Buffer' from 'typing_extensions' (/runner/_work/beam/beam/build/gradleenv/1398941892/lib/python3.8/site-packages/typing_extensions.py)

Note that Buffer was introduced in typing_extensions at 4.6.0, but the version resolved here is 4.5.0.

Successfully installed ...(omitted) typing-extensions-4.5.0 ...(omitted)

During checking the previous log, I found that there was an existing version incompatible problem on typing_extensions even before the recent post commit failures. It looks like a lot of packages depend on a newer version of typing-extensions, while tensorflow requires typing-extensions < 4.6.0.

Collecting typing-extensions<4.6.0,>=3.6.6 (from tensorflow>=2.12.0->-r /runner/_work/beam/beam/sdks/python/apache_beam/ml/inference/tensorflow_tests_requirements.txt (line 19))
Downloading typing_extensions-4.5.0-py3-none-any.whl.metadata (8.5 kB)

...

azure-core 1.30.1 requires typing-extensions>=4.6.0, but you have typing-extensions 4.5.0 which is incompatible.
azure-storage-blob 12.20.0b1 requires typing-extensions>=4.6.0, but you have typing-extensions 4.5.0 which is incompatible.
pydantic 2.7.1 requires typing-extensions>=4.6.1, but you have typing-extensions 4.5.0 which is incompatible.
pydantic-core 2.18.2 requires typing-extensions!=4.7.0,>=4.6.0, but you have typing-extensions 4.5.0 which is incompatible.

However this didn't show up as a PostCommit failure until typeguard was introduced as a dependency from https://github.com/apache/beam/blob/3329edba79f7b99af983b22cf44ab6e77a8ee987/sdks/python/apache_beam/transforms/enrichment_handlers/feast_tests_requirements.txt. (Code submitted on 04/26/2024 2:31PM EDT after the last successful run)

Collecting typeguard>=4.0.0 (from feast[gcp]->-r /runner/_work/beam/beam/sdks/python/apache_beam/transforms/enrichment_handlers/feast_tests_requirements.txt (line 18))

In typeguard, the code tries to import Buffer which causes an import error as shown in the traceback.

Adding @riteshghorse who submitted the above commit to the discussion here.

[riteshghorse]

Thanks for the analysis @shunping! I think I can take this one

[damondouglas]

Numerous errors still showing for PostCommit Python:
https://github.com/apache/beam/actions/runs/9447564413
https://github.com/apache/beam/actions/runs/9447157024
https://github.com/apache/beam/actions/runs/9447191775
https://github.com/apache/beam/actions/runs/9447544394
https://github.com/apache/beam/actions/runs/9447549306

[damccorm]

This seems to have been resolved