[improve][misc] Sync commits from apache into 3.1_ds #305
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Lari Hotari <[email protected]> (cherry picked from commit 702c0b3) (cherry picked from commit 617c110)
…follower (ExtensibleLoadManagerImpl only) (apache#23144) (cherry picked from commit 1b43b9d) (cherry picked from commit 1e1860a)
…c and logging (apache#23142) (cherry picked from commit 9bf714f) (cherry picked from commit 4235a53)
…ers a metadata store error (apache#23153) (cherry picked from commit 2dde403) (cherry picked from commit d406193)
…ParamsAutoClusterFailover (apache#23129) - excluding changes to pulsar-broker/src/test/java/org/apache/pulsar/client/api/NonDurableSubscriptionTest.java PR apache#23168 (cherry picked from commit 06a2f5c) (cherry picked from commit 286a5dc)
nikhil-ctds
force-pushed
the
cherry-picks
branch
2 times, most recently
from
2ebbb7a to
3425830
Compare
apache#23141) (cherry picked from commit c07b158) (cherry picked from commit 69a6232)
Co-authored-by: Ómar Yasin <[email protected]> (cherry picked from commit a025938) (cherry picked from commit 507d402)
… whenever possible (apache#23163) (cherry picked from commit d5ce1ce) (cherry picked from commit bbe67c8)
…not cached (apache#23147) For shadow topics, a `ReadOnlyLedgerHandle` is created to read messages from the source topic when the entry is not cached. However, it leverages the `readAsync` API that validates the `lastAddConfirmed` field (LAC). In `ReadOnlyLedgerHandle`, this field could never be updated, so `readAsync` could fail immediately. See `LedgerHandle#readAsync`: ```java if (lastEntry > lastAddConfirmed) { LOG.error("ReadAsync exception on ledgerId:{} firstEntry:{} lastEntry:{} lastAddConfirmed:{}", ledgerId, firstEntry, lastEntry, lastAddConfirmed); return FutureUtils.exception(new BKReadException()); } ``` This bug is not exposed because: 1. `PulsarMockReadHandle` does not maintain a LAC field. 2. The case for cache miss is never tested. Replace `readAsync` with `readUnconfirmedAsync` and compare the entry range with the `ManagedLedger#getLastConfirmedEntry`. The managed ledger already maintains a `lastConfirmedEntry` to limit the last entry. See `ManagedLedgerImpl#internalReadFromLedger`: ```java Position lastPosition = lastConfirmedEntry; if (ledger.getId() == lastPosition.getLedgerId()) { lastEntryInLedger = lastPosition.getEntryId(); ``` Add `ShadowTopicRealBkTest` to cover two code changes `RangeEntryCacheImpl#readFromStorage` and `EntryCache#asyncReadEntry`. Exceptionally, compare the entry range with the LAC of a ledger handle when it does not exist in the managed ledger. It's because `ReadOnlyManagedLedgerImpl` could read a ledger in another managed ledger. <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> PR in forked repository: BewareMyPower#33 <!-- After opening this PR, the build in apache/pulsar will fail and instructions will be provided for opening a PR in the PR author's forked repository. apache/pulsar pull requests should be first tested in your own fork since the apache/pulsar CI based on GitHub Actions has constrained resources and quota. GitHub Actions provides separate quota for pull requests that are executed in a forked repository. The tests will be run in the forked repository until all PR review comments have been handled, the tests pass and the PR is approved by a reviewer. --> (cherry picked from commit 15b88d2) (cherry picked from commit 14b3672)
… and fix bug in copying (apache#23182) Fixes apache#23173 Fixes apache#23181 See apache#23173 and apache#23181 - copy ordering key to messages sent to retry letter topic and DLQ topic (cherry picked from commit 67fc5b9) (cherry picked from commit c83428f)
srinath-ctds
approved these changes
Aug 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
This PR sync all commits from apache/branch-3.0 into 3.1_ds which are not present till 30th June.
Modifications
Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?
doc-required(If you need help on updating docs, create a doc issue)
no-need-doc(Please explain why)
doc(If this PR contains doc changes)