[yaml] Add use cases for Enrichment transform in YAML #32289

reeba212 · 2024-08-22T18:19:29Z

Added a simple bigtable enrichment example
Added an end to end use case that demonstrates how spanner IO and enrichment transform coupled with other YAML transforms can be used in a real world setting

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

github-actions · 2024-08-22T20:34:34Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label python.
R: @ahmedabu98 for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Polber

Thanks! Added a few comments

sdks/python/apache_beam/io/gcp/bigquery_tools.py

sdks/python/apache_beam/yaml/examples/bigtable_enrichment.yaml

sdks/python/apache_beam/yaml/examples/enrich_spanner_with_bigquery.yaml

Polber · 2024-08-22T20:50:56Z

sdks/python/apache_beam/yaml/examples/enrich_spanner_with_bigquery.yaml

+options:
+  project_id: "apache-beam-testing"


Is this necessary? It is defined in all the transforms, so not sure why it would be required

Polber

Just a couple small changes, thanks!

sdks/python/apache_beam/io/gcp/bigquery_tools.py

sdks/python/apache_beam/yaml/integration_tests.py

github-actions · 2024-09-05T12:14:03Z

Reminder, please take a look at this pr: @damccorm @ahmedabu98

github-actions · 2024-09-09T12:14:09Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @liferoad for label python.
R: @johnjcasey for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

github-actions · 2024-09-16T12:14:20Z

Reminder, please take a look at this pr: @liferoad @johnjcasey

github-actions · 2024-09-19T12:14:11Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @tvalentyn for label python.
R: @Abacn for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

damccorm · 2024-09-19T14:17:17Z

@Polber what are next steps on this PR?

@reeba212 it looks like some precommit checks are failing - would you mind taking a look? At a glance, I see:

the RAT license check is missing - this is because some files are missing the Apache license
Lint/format checks are failing - could you try https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-LintandFormattingChecks
test_bigtable_enrichment_yaml seems to be failing

damccorm · 2024-10-03T15:36:29Z

@Polber @reeba212 friendly reminder to please take a look at this one when you have a chance

End to end use case that demonstrates how spanner IO and enrichment transform coupled with other YAML transforms can be used in the real world

Signed-off-by: Jeffrey Kinard <[email protected]>

fix examples_test

Polber

Left some comments which should fix errors

sdks/python/apache_beam/yaml/examples/testing/examples_test.py

Co-authored-by: Jeff Kinard <[email protected]>

damccorm

Mostly LGTM, but I had one question

damccorm · 2024-12-10T20:07:18Z

sdks/python/apache_beam/io/gcp/bigquery_tools.py

+      client=None,
+      temp_dataset_id=None,
+      temp_table_ref=None,
+      project_id=None):


Do we do anything with this project_id?

From what I remember, running the code without assigning project_id resulted in the following error:
OSError: Project ID was not passed and could not be determined from the environment.

I don't think it is actually getting assigned, though, and other functions take in project_id as parameters. If it is passed in to initialize the object, I think it will be ignored as written

Oh I see. Should I go ahead and remove it, then?

Yeah, lets go ahead and get rid of it - thanks

damccorm · 2024-12-16T14:39:58Z

Failing tests are unrelated to the PR, merging - thanks!

github-actions bot added python yaml labels Aug 22, 2024

reeba212 changed the title ~~Add use-cases for Enrichment transform in YAML~~ [yaml] Add use cases for Enrichment transform in YAML Aug 22, 2024

github-actions bot added io gcp labels Aug 22, 2024

github-actions bot added the Next Action: Reviewers label Aug 22, 2024

Polber suggested changes Aug 22, 2024

View reviewed changes

Polber suggested changes Aug 26, 2024

View reviewed changes

sdks/python/apache_beam/io/gcp/bigquery_tools.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/yaml/integration_tests.py Outdated Show resolved Hide resolved

github-actions bot added the slow-review label Sep 5, 2024

github-actions bot removed the slow-review label Sep 9, 2024

github-actions bot added the slow-review label Sep 16, 2024

github-actions bot removed the slow-review label Sep 19, 2024

damccorm added Next Action: Author and removed Next Action: Reviewers labels Sep 19, 2024

github-actions bot added java go build website infra examples docker extensions runners labels Oct 17, 2024

github-actions bot removed spanner bigtable prism labels Dec 4, 2024

reeba212 force-pushed the spanner-enrichment-use-case branch from dae94d0 to f5d0aa3 Compare December 4, 2024 09:45

github-actions bot removed the docker label Dec 4, 2024

reeba212 and others added 7 commits December 4, 2024 15:28

Create use case for enriching spanner data with bigquery

0dd22b8

End to end use case that demonstrates how spanner IO and enrichment transform coupled with other YAML transforms can be used in the real world

Create example for bigtable enrichment

c0ea43a

Add project_id parameter to BigQueryWrapper

78da902

minor changes

ae0d972

remove project id being passed into bigquery wrapper

85eda70

add license

29507be

add expected blocks

b4aa22f

reeba212 force-pushed the spanner-enrichment-use-case branch from f5d0aa3 to b4aa22f Compare December 4, 2024 10:01

reeba212 and others added 4 commits December 4, 2024 15:51

Update examples_test.py

a138d42

Update examples_test.py

7ae4284

fix formatting

fad1f7f

fix examples_test

0f32323

Signed-off-by: Jeffrey Kinard <[email protected]>

Polber mentioned this pull request Dec 6, 2024

fix examples_test reeba212/beam#1

Merged

3 tasks

Merge pull request #1 from Polber/spanner-enrichment-use-case

5988a23

fix examples_test

Polber suggested changes Dec 9, 2024

View reviewed changes

Apply suggestions from code review

bb511e9

Co-authored-by: Jeff Kinard <[email protected]>

Polber approved these changes Dec 10, 2024

View reviewed changes

damccorm reviewed Dec 10, 2024

View reviewed changes

reeba212 added 2 commits December 13, 2024 23:36

Update bigquery_tools.py

1a2e9da

Update bigquery_tools.py

cd136c9

github-actions bot removed io gcp labels Dec 13, 2024

damccorm approved these changes Dec 16, 2024

View reviewed changes

damccorm merged commit f77efba into apache:master Dec 16, 2024
86 of 92 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[yaml] Add use cases for Enrichment transform in YAML #32289

[yaml] Add use cases for Enrichment transform in YAML #32289

reeba212 commented Aug 22, 2024

github-actions bot commented Aug 22, 2024

Polber left a comment

Polber Aug 22, 2024

Polber left a comment

github-actions bot commented Sep 5, 2024

github-actions bot commented Sep 9, 2024

github-actions bot commented Sep 16, 2024

github-actions bot commented Sep 19, 2024

damccorm commented Sep 19, 2024

damccorm commented Oct 3, 2024

Polber left a comment

damccorm left a comment

damccorm Dec 10, 2024

reeba212 Dec 13, 2024

damccorm Dec 13, 2024

reeba212 Dec 13, 2024

damccorm Dec 13, 2024

damccorm commented Dec 16, 2024

[yaml] Add use cases for Enrichment transform in YAML #32289

[yaml] Add use cases for Enrichment transform in YAML #32289

Conversation

reeba212 commented Aug 22, 2024

GitHub Actions Tests Status (on master branch)

github-actions bot commented Aug 22, 2024

Polber left a comment

Choose a reason for hiding this comment

Polber Aug 22, 2024

Choose a reason for hiding this comment

Polber left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 5, 2024

github-actions bot commented Sep 9, 2024

github-actions bot commented Sep 16, 2024

github-actions bot commented Sep 19, 2024

damccorm commented Sep 19, 2024

damccorm commented Oct 3, 2024

Polber left a comment

Choose a reason for hiding this comment

damccorm left a comment

Choose a reason for hiding this comment

damccorm Dec 10, 2024

Choose a reason for hiding this comment

reeba212 Dec 13, 2024

Choose a reason for hiding this comment

damccorm Dec 13, 2024

Choose a reason for hiding this comment

reeba212 Dec 13, 2024

Choose a reason for hiding this comment

damccorm Dec 13, 2024

Choose a reason for hiding this comment

damccorm commented Dec 16, 2024