Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python] Fail WindowInto when there are no inputs to it #28946

Merged
merged 4 commits into from
Oct 18, 2023

Conversation

riteshghorse
Copy link
Contributor

@riteshghorse riteshghorse commented Oct 11, 2023

It has been observed that pipelines get stuck when there are no input PCollections to a beam.WindowInto. No logs are observed in the console and the only option left is to force terminate jobs. This PR catches that case and hard fails the pipeline at submission time.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@codecov
Copy link

codecov bot commented Oct 11, 2023

Codecov Report

Merging #28946 (bdb2298) into master (2bfcb9f) will decrease coverage by 33.78%.
Report is 80 commits behind head on master.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           master   #28946       +/-   ##
===========================================
- Coverage   72.16%   38.39%   -33.78%     
===========================================
  Files         686      686               
  Lines      101560   101630       +70     
===========================================
- Hits        73294    39017    -34277     
- Misses      26687    61034    +34347     
  Partials     1579     1579               
Flag Coverage Δ
python 30.00% <0.00%> (-52.62%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
sdks/python/apache_beam/runners/common.py 68.21% <0.00%> (-20.50%) ⬇️

... and 312 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@riteshghorse
Copy link
Contributor Author

Run Python 3.8 PostCommit

@riteshghorse riteshghorse marked this pull request as ready for review October 12, 2023 13:18
@riteshghorse riteshghorse marked this pull request as draft October 12, 2023 13:18
@riteshghorse riteshghorse marked this pull request as ready for review October 12, 2023 13:19
@riteshghorse
Copy link
Contributor Author

R: @tvalentyn

@riteshghorse riteshghorse changed the title fail windowInto when there are no inputs to it [Python] Fail WindowInto when there are no inputs to it Oct 12, 2023
@github-actions
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@robertwb
Copy link
Contributor

FWIW, I think this should be fixed in the service rather than adding a special case to the SDK (Python only, which won't be out for 6 weeks or so).

@robertwb
Copy link
Contributor

A good place to validate this might be https://github.com/apache/beam/blob/release-2.51.0/sdks/python/apache_beam/runners/common.py#L1895

We could have a mapping of known URNs -> expected input/output cardinality (if any), and flag violations. (We could ensure the inputs/outputs actually exist.)

p | beam.WindowInto(...) should also be an error right there (checked in WindowInto's expand method).

sdks/python/apache_beam/runners/common.py Outdated Show resolved Hide resolved
sdks/python/apache_beam/runners/common.py Outdated Show resolved Hide resolved
@robertwb robertwb merged commit dbc5c4e into apache:master Oct 18, 2023
61 of 73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants