Add various utility meta-transforms to Beam. #32445

robertwb · 2024-09-12T19:48:49Z

These were inspired by some discussions at the Beam summit.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

github-actions · 2024-09-12T20:34:54Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @liferoad for label python.
R: @Abacn for label java.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

liferoad · 2024-09-12T20:45:20Z

Shall we add this to https://beam.apache.org/documentation/programming-guide/#flatten as one of core PTransform?

robertwb · 2024-09-17T22:13:18Z

Added a note about the Flatten alternative. I don't think Tee is important/core enough to add to the guide itself, but have been thinking it might be good to do a blog post calling out these and other utility transforms (like BatchElements).

hjtran · 2024-09-26T20:50:32Z

These look really convenient. Especially for pipelines that might write out intermediate results or merge older pcollections (something the schrodinger use cases do a lot).

Added a note about the Flatten alternative. I don't think Tee is important/core enough to add to the guide itself, but have been thinking it might be good to do a blog post calling out these and other utility transforms (like BatchElements).

I personally discover useful transforms through the beam transform catalog so it'd be nice if some examples were included there.

github-actions · 2024-10-04T12:14:17Z

Reminder, please take a look at this pr: @liferoad @Abacn

github-actions · 2024-10-12T12:14:04Z

Reminder, please take a look at this pr: @liferoad @Abacn

github-actions · 2024-10-16T12:14:17Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @tvalentyn for label python.
R: @damondouglas for label java.
R: @melap for label website.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Flatten.java

tvalentyn · 2024-10-16T16:55:00Z

sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Flatten.java

+
+    @Override
+    public String getKindString() {
+      return "Flatten.With";


Merge.With might be a possible alternative name. but maybe it adds more confusion since we have a pre-existing Flatten already for a similar concept.

Yeah, I chose this name because it is literally syntactic sugar for the same primitive Flatten operation. (Personally, I'd prefer disjoint union, but that's probably to obscure let alone too late to change now...)

sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Flatten.java

tvalentyn · 2024-10-16T18:39:26Z

sdks/python/apache_beam/examples/snippets/snippets.py

+    pcoll1 = partitioned[0]
+    pcoll2 = partitioned[1]
+    pcoll3 = partitioned[2]
+    SomeTransform = lambda: beam.Map(lambda x: x)


Should we have other example for merging with a transform output? feel free to create a bug to add it. examples are just important as having the capability, so i think we should highlight these everywhere (beam playground, snippets, website docs, etc). Can be with follow up /starter bugs if you don't have time to do all that in one change.

Added this example, filed #32840 for follow-up. It would be good to think about how we could structure things to further reduce redundancy between these various forms of documentation.

sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Tee.java

Co-authored-by: tvalentyn <[email protected]>

robertwb

Thanks for taking a look. I added one more example, but filed an issue for further documentation in the interest of not blocking things.

robertwb · 2024-10-17T16:11:45Z

sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Flatten.java

+
+    @Override
+    public String getKindString() {
+      return "Flatten.With";


Yeah, I chose this name because it is literally syntactic sugar for the same primitive Flatten operation. (Personally, I'd prefer disjoint union, but that's probably to obscure let alone too late to change now...)

robertwb · 2024-10-17T16:53:48Z

sdks/python/apache_beam/examples/snippets/snippets.py

+    pcoll1 = partitioned[0]
+    pcoll2 = partitioned[1]
+    pcoll3 = partitioned[2]
+    SomeTransform = lambda: beam.Map(lambda x: x)


Added this example, filed #32840 for follow-up. It would be good to think about how we could structure things to further reduce redundancy between these various forms of documentation.

tvalentyn · 2024-10-17T19:24:09Z

I personally discover useful transforms through the beam transform catalog so it'd be nice if some examples were included there.

@robertwb do we need an issue or commit for this as well?

PTAL at website failures & please look at stage website content before merge to see if the changes reflect your intent. Link should be available in the Summary tab of the Stage_GCS GithubAction run.

LGTM otherwise, thanks!

robertwb · 2024-10-17T19:48:43Z

Thanks. Looking into the website failures...

robertwb · 2024-10-17T19:49:10Z

I personally discover useful transforms through the beam transform catalog so it'd be nice if some examples were included there.

@robertwb do we need an issue or commit for this as well?

Update the description to mention both.

github-actions bot added python java labels Sep 12, 2024

github-actions bot added the Next Action: Reviewers label Sep 12, 2024

github-actions bot added website examples labels Sep 17, 2024

github-actions bot added website examples and removed website examples labels Sep 17, 2024

robertwb force-pushed the tee branch 2 times, most recently from 0be36ca to 58e0229 Compare September 17, 2024 23:21

robertwb added 2 commits September 20, 2024 14:45

Add various utility meta-transforms to Beam.

81a50f2

Add note about FlattenWith to the documentation.

71d97b6

robertwb force-pushed the tee branch from 58e0229 to 71d97b6 Compare September 20, 2024 21:45

Fix checkstyle rule.

efcf1ba

github-actions bot added slow-review and removed slow-review labels Oct 4, 2024

github-actions bot added the slow-review label Oct 12, 2024

github-actions bot removed the slow-review label Oct 16, 2024

tvalentyn reviewed Oct 16, 2024

View reviewed changes

robertwb and others added 3 commits October 17, 2024 09:14

Apply suggestions from code review

acd52f9

Co-authored-by: tvalentyn <[email protected]>

Add example of FlattenWith accepting a root PTransform.

177c9ab

Merge branch 'master' into tee

7c6ee3e

robertwb mentioned this pull request Oct 17, 2024

[Feature Request]: Add examples of FlattenWith and Tee to the playground and transform catalog. #32840

Open

17 tasks

robertwb commented Oct 17, 2024

View reviewed changes

Fix doc references.

621cdfb

robertwb force-pushed the tee branch from 907a1a6 to 621cdfb Compare October 17, 2024 23:25

robertwb merged commit 3b839d1 into apache:master Oct 18, 2024
118 of 120 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add various utility meta-transforms to Beam. #32445

Add various utility meta-transforms to Beam. #32445

robertwb commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

liferoad commented Sep 12, 2024

robertwb commented Sep 17, 2024

hjtran commented Sep 26, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 12, 2024

github-actions bot commented Oct 16, 2024

tvalentyn Oct 16, 2024

robertwb Oct 17, 2024

tvalentyn Oct 16, 2024

robertwb Oct 17, 2024

robertwb left a comment

robertwb Oct 17, 2024

robertwb Oct 17, 2024

tvalentyn commented Oct 17, 2024

robertwb commented Oct 17, 2024

robertwb commented Oct 17, 2024

Add various utility meta-transforms to Beam. #32445

Add various utility meta-transforms to Beam. #32445

Conversation

robertwb commented Sep 12, 2024

GitHub Actions Tests Status (on master branch)

github-actions bot commented Sep 12, 2024

liferoad commented Sep 12, 2024

robertwb commented Sep 17, 2024

hjtran commented Sep 26, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 12, 2024

github-actions bot commented Oct 16, 2024

tvalentyn Oct 16, 2024

Choose a reason for hiding this comment

robertwb Oct 17, 2024

Choose a reason for hiding this comment

tvalentyn Oct 16, 2024

Choose a reason for hiding this comment

robertwb Oct 17, 2024

Choose a reason for hiding this comment

robertwb left a comment

Choose a reason for hiding this comment

robertwb Oct 17, 2024

Choose a reason for hiding this comment

robertwb Oct 17, 2024

Choose a reason for hiding this comment

tvalentyn commented Oct 17, 2024

robertwb commented Oct 17, 2024

robertwb commented Oct 17, 2024