-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Export gcs options #30772
Export gcs options #30772
Conversation
5f0a7fd
to
0928146
Compare
Run Java PreCommit |
Run Java_IOs_Direct PreCommit |
0928146
to
9cb95de
Compare
fb6e5db
to
b4cc808
Compare
Run Java PreCommit |
Run Java_IOs_Direct PreCommit |
b4cc808
to
5cfe6f3
Compare
Run Java PreCommit |
5cfe6f3
to
91ffa17
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #30772 +/- ##
============================================
- Coverage 71.47% 70.85% -0.62%
- Complexity 0 2983 +2983
============================================
Files 710 1062 +352
Lines 104815 132683 +27868
Branches 0 3230 +3230
============================================
+ Hits 74915 94012 +19097
- Misses 28268 35576 +7308
- Partials 1632 3095 +1463
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Run Java_IOs_Direct PreCommit |
1 similar comment
Run Java_IOs_Direct PreCommit |
Test seems flaky, will try rerunning again: |
Run Java_IOs_Direct PreCommit |
Checks are passing, the one failing is for parsing test results (saying the file is too large: https://github.com/apache/beam/pull/30772/checks?check_run_id=23251024301) |
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
assign set of reviewers |
Assigning reviewers. If you would like to opt out of this review, comment R: @bvolpato for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Reminder, please take a look at this pr: @bvolpato |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #30772 +/- ##
============================================
- Coverage 71.47% 70.85% -0.62%
- Complexity 0 2983 +2983
============================================
Files 710 1062 +352
Lines 104815 132683 +27868
Branches 0 3230 +3230
============================================
+ Hits 74915 94012 +19097
- Misses 28268 35576 +7308
- Partials 1632 3095 +1463
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Reminder, please take a look at this pr: @bvolpato |
R: @scwhittle |
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
@@ -175,6 +175,9 @@ public static <UserT, DestinationT, OutputT> WriteFiles<UserT, DestinationT, Out | |||
.setSkipIfEmpty(false) | |||
.setBadRecordErrorHandler(new DefaultErrorHandler<>()) | |||
.setBadRecordRouter(BadRecordRouter.THROWING_ROUTER) | |||
.setFileTriggeringRecordCount(FILE_TRIGGERING_RECORD_COUNT) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
instead of repeating these everywhere we create the builder how about creating a static create method that constructs the builder and adds defaults instead of calling the builder constructor directly. Could unify the other defaults as well.
pseudocode:
in AutoValue_WriteFiles.Builder
static Builder create() {
return new AutoValue_WriteFiles.Builder<UserT, DestinationT, OutputT>.Builder()
.setFileTriggeringByteCount(FILE_TRIGGERING_BYTE_COUNT) // 64MiB as of now
}
and make constructor private
I believe that you can override th builder constructor to set the defaults. That seems better in that it will have a single location instead of duplicated across different files.
@Naireen Friendly ping, is this still relevant? |
Yes this is, sorry this fell through the cracks, will address your comments and rebase this PR sometimes next week. |
Actually, I see that #31617 went in last month, so I'm going to close this! |
Allow Customers to pass GIB.WSK parameters for the write File transform.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123
), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>
instead.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.