fix: a rare race condition in the row merger #1939
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This would manifest as a hang when iterating over a ServerStream from ReadRows Change-Id: I74533c6714b40a68ec0ef81dadac747e10bee39d
product-auto-label
bot
added
size: m
steveniemitz
approved these changes
Sep 28, 2023
gcf-merge-on-green bot
pushed a commit
that referenced
this pull request
Sep 29, 2023
🤖 I have created a release *beep* *boop* --- ## [2.27.3](https://togithub.com/googleapis/java-bigtable/compare/v2.27.2...v2.27.3) (2023-09-29) ### Bug Fixes * A rare race condition in the row merger ([#1939](https://togithub.com/googleapis/java-bigtable/issues/1939)) ([fccd710](https://togithub.com/googleapis/java-bigtable/commit/fccd7100578da3e2ee2a315c6e956c06dbf41668)) ### Dependencies * Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.27 ([#1919](https://togithub.com/googleapis/java-bigtable/issues/1919)) ([56d6b40](https://togithub.com/googleapis/java-bigtable/commit/56d6b40b9f874ec4ad78618c1cf0d365ddc9276b)) * Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.27 ([#1920](https://togithub.com/googleapis/java-bigtable/issues/1920)) ([ca1dd5b](https://togithub.com/googleapis/java-bigtable/commit/ca1dd5b258c7fa5cfd90811dc8b9492c19d1f104)) * Update gapic-generator-java to 2.26.0 ([#1936](https://togithub.com/googleapis/java-bigtable/issues/1936)) ([15cd486](https://togithub.com/googleapis/java-bigtable/commit/15cd4868ff807513914095a3758134eaa14f0ea3)) --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
4 tasks
mutianf
pushed a commit
to mutianf/java-bigtable
that referenced
this pull request
Nov 3, 2023
* fix: a rare race condition in the row merger This would manifest as a hang when iterating over a ServerStream from ReadRows Change-Id: I74533c6714b40a68ec0ef81dadac747e10bee39d * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
mutianf
added a commit
that referenced
this pull request
Nov 6, 2023
* fix: a rare race condition in the row merger (#1939) * fix: a rare race condition in the row merger This would manifest as a hang when iterating over a ServerStream from ReadRows Change-Id: I74533c6714b40a68ec0ef81dadac747e10bee39d * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Igor Bernstein <[email protected]> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
mutianf
pushed a commit
to mutianf/java-bigtable
that referenced
this pull request
Nov 8, 2023
* fix: a rare race condition in the row merger This would manifest as a hang when iterating over a ServerStream from ReadRows Change-Id: I74533c6714b40a68ec0ef81dadac747e10bee39d * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
mutianf
added a commit
that referenced
this pull request
Nov 8, 2023
* fix: a rare race condition in the row merger (#1939) * fix: a rare race condition in the row merger This would manifest as a hang when iterating over a ServerStream from ReadRows Change-Id: I74533c6714b40a68ec0ef81dadac747e10bee39d * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * chore: Fix flaky metrics tests (#1865) This fixes a few flaky unit tests that relied on `Thread.sleep` to ensure that all metrics processing was done. Rather than using `Thread.sleep`, we can instead use an inline event queue in the OpenCensus stats component to execute all work inline, removing the need to wait for anything to finish. --------- Co-authored-by: Igor Bernstein <[email protected]> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Steven Niemitz <[email protected]>
igorbernstein2
added a commit
to igorbernstein2/sdk-platform-java
that referenced
this pull request
Aug 29, 2024
Sometimes an operation can get stuck indefinitely. The underlying reasons can vary significantly: - the underlying attempt rpc can get stuck due to a bug in grpc (ie grpc/grpc-java#11026) - the operation can get stuck in layers above gax: googleapis/java-bigtable#1939 - or it can get stuck in gax itself (dont have a pointer handy) Guava futures provide some observability for ListenableFutures, but in creating the custom ApiFutures in gax, we lose that functionality. This PR sprinkles a few to toString to allow callers to inspect the internal state of the operation. For example with these changes, the toString() of the future returned from bigtableDataClient.mutateRows() changes from TransformFuture@652ce654[status=PENDING, info=[inputFuture=[com.google.api.core.ApiFutureToListenableFuture@522ba524], function=[com.google.api.core.ApiFutures$ApiFunctionToGuavaFunction@29c5ee1d]]] to ListenableFutureToApiFuture{delegate=TransformFuture@7ac9af2a[status=PENDING, info=[inputFuture=[ApiFutureToListenableFuture{apiFuture=CallbackChainRetryingFuture{super=com.google.api.gax.retrying.CallbackChainRetryingFuture@7bb004b8[status=PENDING], latestCompletedAttemptResult=null, attemptResult=null, attemptSettings=TimedAttemptSettings{globalSettings=RetrySettings{totalTimeout=PT10M, initialRetryDelay=PT0.01S, retryDelayMultiplier=2.0, maxRetryDelay=PT1M, maxAttempts=0, jittered=true, initialRpcTimeout=PT1M, rpcTimeoutMultiplier=1.0, maxRpcTimeout=PT1M}, retryDelay=PT0S, rpcTimeout=PT1M, randomizedRetryDelay=PT0S, attemptCount=0, overallAttemptCount=0, firstAttemptStartTimeNanos=635709620001791}}}], function=[com.google.api.core.ApiFutures$ApiFunctionToGuavaFunction@652ce654]]]} This allows us to reason about whats stuck. I'm working another PR that will add a close(timeout) to the Batcher that will use this functionality to identify why batcher.close() timed out
4 tasks
blakeli0
pushed a commit
to googleapis/sdk-platform-java
that referenced
this pull request
Aug 30, 2024
Sometimes an operation can get stuck indefinitely. The underlying reasons can vary significantly: - the underlying attempt rpc can get stuck due to a bug in grpc (ie grpc/grpc-java#11026) - the operation can get stuck in layers above gax: googleapis/java-bigtable#1939 - or it can get stuck in gax itself (dont have a pointer handy) Guava futures provide some observability for ListenableFutures, but in creating the custom ApiFutures in gax, we lose that functionality. This PR sprinkles a few to toString to allow callers to inspect the internal state of the operation. For example with these changes, the toString() of the future returned from bigtableDataClient.mutateRows() changes from > TransformFuture@652ce654[status=PENDING, info=[inputFuture=[com.google.api.core.ApiFutureToListenableFuture@522ba524], function=[com.google.api.core.ApiFutures$ApiFunctionToGuavaFunction@29c5ee1d]]] to > ListenableFutureToApiFuture{delegate=TransformFuture@7ac9af2a[status=PENDING, info=[inputFuture=[ApiFutureToListenableFuture{apiFuture=CallbackChainRetryingFuture{super=com.google.api.gax.retrying.CallbackChainRetryingFuture@7bb004b8[status=PENDING], latestCompletedAttemptResult=null, attemptResult=null, attemptSettings=TimedAttemptSettings{globalSettings=RetrySettings{totalTimeout=PT10M, initialRetryDelay=PT0.01S, retryDelayMultiplier=2.0, maxRetryDelay=PT1M, maxAttempts=0, jittered=true, initialRpcTimeout=PT1M, rpcTimeoutMultiplier=1.0, maxRpcTimeout=PT1M}, retryDelay=PT0S, rpcTimeout=PT1M, randomizedRetryDelay=PT0S, attemptCount=0, overallAttemptCount=0, firstAttemptStartTimeNanos=635709620001791}}}], function=[com.google.api.core.ApiFutures$ApiFunctionToGuavaFunction@652ce654]]]} This allows us to reason about whats stuck. I'm working another PR that will add a close(timeout) to the Batcher that will use this functionality to identify why batcher.close() timed out
ldetmer
pushed a commit
to googleapis/sdk-platform-java
that referenced
this pull request
Sep 17, 2024
Sometimes an operation can get stuck indefinitely. The underlying reasons can vary significantly: - the underlying attempt rpc can get stuck due to a bug in grpc (ie grpc/grpc-java#11026) - the operation can get stuck in layers above gax: googleapis/java-bigtable#1939 - or it can get stuck in gax itself (dont have a pointer handy) Guava futures provide some observability for ListenableFutures, but in creating the custom ApiFutures in gax, we lose that functionality. This PR sprinkles a few to toString to allow callers to inspect the internal state of the operation. For example with these changes, the toString() of the future returned from bigtableDataClient.mutateRows() changes from > TransformFuture@652ce654[status=PENDING, info=[inputFuture=[com.google.api.core.ApiFutureToListenableFuture@522ba524], function=[com.google.api.core.ApiFutures$ApiFunctionToGuavaFunction@29c5ee1d]]] to > ListenableFutureToApiFuture{delegate=TransformFuture@7ac9af2a[status=PENDING, info=[inputFuture=[ApiFutureToListenableFuture{apiFuture=CallbackChainRetryingFuture{super=com.google.api.gax.retrying.CallbackChainRetryingFuture@7bb004b8[status=PENDING], latestCompletedAttemptResult=null, attemptResult=null, attemptSettings=TimedAttemptSettings{globalSettings=RetrySettings{totalTimeout=PT10M, initialRetryDelay=PT0.01S, retryDelayMultiplier=2.0, maxRetryDelay=PT1M, maxAttempts=0, jittered=true, initialRpcTimeout=PT1M, rpcTimeoutMultiplier=1.0, maxRpcTimeout=PT1M}, retryDelay=PT0S, rpcTimeout=PT1M, randomizedRetryDelay=PT0S, attemptCount=0, overallAttemptCount=0, firstAttemptStartTimeNanos=635709620001791}}}], function=[com.google.api.core.ApiFutures$ApiFunctionToGuavaFunction@652ce654]]]} This allows us to reason about whats stuck. I'm working another PR that will add a close(timeout) to the Batcher that will use this functionality to identify why batcher.close() timed out
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This would manifest as a hang when iterating over a ServerStream from ReadRows
Change-Id: I74533c6714b40a68ec0ef81dadac747e10bee39d
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #<issue_number_goes_here> ☕️
If you write sample code, please follow the samples format.