Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propagate annotations of CombinePerKey transforms to resulting component transforms #28489

Merged
merged 4 commits into from
Sep 18, 2023

Conversation

hjtran
Copy link
Contributor

@hjtran hjtran commented Sep 15, 2023

Issue:

If somehow you have an annotation(s) on a CombinePerKey transform (e.g. because you subclass it and add annotations or a runner adds annotations), the annotations end up disappearing after getting translated by lifted_stages.

Considerations:

I don't think it's super obvious what the correct behavior is here. It's a bit surprising the annotation disappears but it may also be surprising for someone who meant to signal some behavior for one transform to suddenly have it for the four resulting component transforms. I put up this PR in case it does make sense to propagate the annotations since for our use case it simplifies things a lot

Context:

We're writing a runner and we'd like to be able to save the top level user labels that users assign to their transforms so we can nicely log which user transform is being run at every phase. The most obvious way to me to do this was to create some custom annotations and then search for them at phase execution time. I was surprised when the CPK annotations were missing. Open to other ways to do this though

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@hjtran
Copy link
Contributor Author

hjtran commented Sep 15, 2023

R: @robertwb

@github-actions
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

Copy link
Contributor

@robertwb robertwb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. LGTM once yapf/lint are happy.

@codecov
Copy link

codecov bot commented Sep 16, 2023

Codecov Report

Merging #28489 (b3a8e15) into master (38720a3) will increase coverage by 0.00%.
Report is 4 commits behind head on master.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #28489   +/-   ##
=======================================
  Coverage   72.22%   72.23%           
=======================================
  Files         684      684           
  Lines      100856   100855    -1     
=======================================
+ Hits        72846    72848    +2     
+ Misses      26434    26431    -3     
  Partials     1576     1576           
Flag Coverage Δ
python 82.82% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
.../runners/portability/fn_api_runner/translations.py 96.79% <ø> (ø)

... and 6 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@hjtran
Copy link
Contributor Author

hjtran commented Sep 16, 2023

Thanks. LGTM once yapf/lint are happy.

Done

@robertwb robertwb merged commit 881c597 into apache:master Sep 18, 2023
77 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants