Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updates the Transform Service to accept Python extra packages through the Java API #28783

Merged
merged 3 commits into from
Oct 11, 2023
Merged

Updates the Transform Service to accept Python extra packages through the Java API #28783

merged 3 commits into from
Oct 11, 2023

Conversation

chamikaramj
Copy link
Contributor

Also performs following fixes.

  • Transform service controller waits for expansion services to start before performing expansion or discovery operations.
  • When expansion requests fail, transform service controller sends back a detailed error with errors from all expansion services instead of just sending the last failure.

Issue is #26833.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@chamikaramj
Copy link
Contributor Author

R: @Abacn

@chamikaramj chamikaramj requested a review from Abacn October 3, 2023 00:38
@github-actions
Copy link
Contributor

github-actions bot commented Oct 3, 2023

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@chamikaramj chamikaramj force-pushed the transform_service_dependencies branch from eb84ec1 to 479b15f Compare October 3, 2023 00:42
@Abacn Abacn requested review from Abacn and removed request for Abacn October 3, 2023 20:48
Abacn
Abacn reviewed Oct 4, 2023
View reviewed changes
Copy link
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the change looks good - noticed that the unit test is not get exercised

...er/src/main/java/org/apache/beam/sdk/transformservice/launcher/TransformServiceLauncher.java Outdated Show resolved Hide resolved
...rc/test/java/org/apache/beam/sdk/transformservice/launcher/TransformServiceLauncherTest.java Show resolved Hide resolved
...a/transform-service/src/main/java/org/apache/beam/sdk/transformservice/ExpansionService.java Outdated Show resolved Hide resolved
...a/transform-service/src/main/java/org/apache/beam/sdk/transformservice/ExpansionService.java Outdated
try {
String url = endpoint.getUrl();
int portIndex = url.lastIndexOf(":");
int port = portIndex > 0 ? Integer.parseInt(url.substring(portIndex + 1)) : 80;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is default port 80 documented somewhere / rely on some other components ? If I remember correctly the port for expansion service is dynamically signed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM. Have you been able to check this comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Updated.

@github-actions github-actions bot added the build label Oct 9, 2023
@codecov
Copy link

codecov bot commented Oct 9, 2023 •
edited
Loading

Codecov Report

Merging #28783 (0834573) into master (5e38dec) will decrease coverage by 0.07%.
Report is 108 commits behind head on master.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master   #28783      +/-   ##
==========================================
- Coverage   72.21%   72.15%   -0.07%     
==========================================
  Files         684      686       +2     
  Lines      101241   101562     +321     
==========================================
+ Hits        73116    73281     +165     
- Misses      26546    26702     +156     
  Partials     1579     1579
Flag Coverage Δ
python 82.61% <0.00%> (-0.16%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...on/apache_beam/utils/transform_service_launcher.py 23.07% <0.00%> (-1.37%) ⬇️

... and 28 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@chamikaramj chamikaramj force-pushed the transform_service_dependencies branch 2 times, most recently from 6300a4d to 59238a3 Compare October 9, 2023 04:20
@chamikaramj chamikaramj force-pushed the transform_service_dependencies branch from 59238a3 to dcfe3d6 Compare October 9, 2023 05:27
chamikaramj
chamikaramj commented Oct 9, 2023
View reviewed changes
Copy link
Contributor Author

@chamikaramj chamikaramj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. PTAL.

...a/transform-service/src/main/java/org/apache/beam/sdk/transformservice/ExpansionService.java Outdated Show resolved Hide resolved
...er/src/main/java/org/apache/beam/sdk/transformservice/launcher/TransformServiceLauncher.java Outdated Show resolved Hide resolved
...rc/test/java/org/apache/beam/sdk/transformservice/launcher/TransformServiceLauncherTest.java Show resolved Hide resolved
@chamikaramj
Copy link
Contributor Author

Run Java PreCommit

@chamikaramj
Copy link
Contributor Author

Run Java_GCP_IO_Direct PreCommit

@chamikaramj
Copy link
Contributor Author

Run Python_Transforms PreCommit

Abacn
Abacn approved these changes Oct 10, 2023
View reviewed changes
Copy link
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM

before merge, check that github action jobs are also completed - #28909

@chamikaramj
Copy link
Contributor Author

Run Java PreCommit

@chamikaramj
Copy link
Contributor Author

Run TransformService_Direct PostCommit

@chamikaramj
Copy link
Contributor Author

Run Portable_Python PreCommit

@chamikaramj
Copy link
Contributor Author

Run Python_PVR_Flink PreCommit

@chamikaramj
Copy link
Contributor Author

Thanks. All tests are passing.

@chamikaramj chamikaramj merged commit 99c87a2 into apache:master Oct 11, 2023
95 of 96 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Abacn Abacn Abacn approved these changes

Assignees
No one assigned
Labels
build core extensions java python runners
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@chamikaramj @Abacn