Add option to adjust ReadChangeStream API timeout to allow longer timeout for longer checkpoint durations #33001
Conversation
|
@jackdingilian for Bigtable review |
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
Outdated
Show resolved
Hide resolved
…eout for longer checkpoint durations Change-Id: Ia61388bed5997b7138b15a014d6370d49cbf5277
tonytanger
force-pushed
the
bigtable_add_display_data
branch
from
fcd7425 to
2486314
Compare
Change-Id: If34866637045314dfeef1807122cc34b97cc5ddc
|
Assigning reviewers. If you would like to opt out of this review, comment R: @Abacn for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
Reminder, please take a look at this pr: @Abacn @johnjcasey @mutianf |
| @@ -52,6 +52,7 @@ public class BigtableChangeStreamAccessor implements AutoCloseable { | |||
| // Create one bigtable data/admin client per bigtable config (project/instance/table/app profile) | |||
| private static final ConcurrentHashMap<BigtableConfig, BigtableChangeStreamAccessor> | |||
| bigtableAccessors = new ConcurrentHashMap<>(); | |||
| private static Duration readChangeStreamTimeout = Duration.ofSeconds(15); | |||
There was a problem hiding this comment.
should this be null? DEFAULT_READ_CHANGE_STREAM_TIMEOUT already defines the default.
There was a problem hiding this comment.
Sorry for the late reply.
I updated the location of the default value into the Accessor. This makes it more clear what the default value is and where it is set. BigtableIO, instead of setting it, will override the default timestamp if necessary.
|
I see it get approved and there is a minor question. Will merge once it is resolved |
|
Reminder, please take a look at this pr: @Abacn @johnjcasey @mutianf |
|
waiting on author |
github-actions
bot
added
Next Action: Author
and removed
Next Action: Reviewers
slow-review
labels
…on about where default is set Change-Id: Ia76ff2e3ba4f6a91299e907ec404b720e900c76f
Change-Id: I89ba1d48742d5204168249d632accf86d68e140c
|
Reminder, please take a look at this pr: @Abacn @johnjcasey @mutianf |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @damondouglas for label java. Available commands:
|
This is useful if the runner's checkpoint duration is longer than 15s which is the default we set for ReadChangeStream API. If the runner's checkpoint duration is longer than 15s, the ReadChangeStream request will get deadline exceeded error before the runner initiated. This will cause the run to fail and return to the previous checkpoint.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.