kafka replay speed: fall back to franz-go for ongoing fetching #9500
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dimitarvdimitrov
force-pushed
the
dimitar/ingest/replay-speed/use-franz-go-for-ongoing-fetching
branch
from
f9627dd to
bd2a078
Compare
dimitarvdimitrov
commented
Oct 2, 2024
| @@ -1440,6 +1440,125 @@ func TestPartitionReader_ConsumeAtStartup(t *testing.T) { | |||
| }) | |||
| } | |||
| }) | |||
|
|
|||
| t.Run("should read target lag and then consume more records after switching to 0 ongoing concurrency if position=start, startup_fetch_concurrency=2, ongoing_fetch_concurrency=0", func(t *testing.T) { | |||
There was a problem hiding this comment.
this is effectively a copy of another test case above but with configured startup and ongoing concurrency
t.Run("should consume partition from start if position=start, and wait until target lag is honored, and then consume some records after lag is honored", func(t *testing.T) {
bd2a078 to
d8dcfc7
Compare
gotjosh
approved these changes
Oct 2, 2024
Striking the right configuration for ongoing fetching depends a lot on the characteristics of cluster. franz-go is better at adaptive concurrency, so we fall back to it until we've implemented adaptive concurrency and/or records-per-fetch. Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
dimitarvdimitrov
force-pushed
the
dimitar/ingest/replay-speed/use-franz-go-for-ongoing-fetching
branch
from
de335b8 to
e376f6b
Compare
dimitarvdimitrov
deleted the
dimitar/ingest/replay-speed/use-franz-go-for-ongoing-fetching
branch
grafanabot
pushed a commit
that referenced
this pull request
Oct 3, 2024
* kafka replay speed: fall back to franz-go for ongoing fetching Striking the right configuration for ongoing fetching depends a lot on the characteristics of cluster. franz-go is better at adaptive concurrency, so we fall back to it until we've implemented adaptive concurrency and/or records-per-fetch. Signed-off-by: Dimitar Dimitrov <[email protected]> * Remove redundant panic Signed-off-by: Dimitar Dimitrov <[email protected]> --------- Signed-off-by: Dimitar Dimitrov <[email protected]> (cherry picked from commit 3ea4cf4)
dimitarvdimitrov
added a commit
that referenced
this pull request
Oct 3, 2024
#9510) * kafka replay speed: fall back to franz-go for ongoing fetching Striking the right configuration for ongoing fetching depends a lot on the characteristics of cluster. franz-go is better at adaptive concurrency, so we fall back to it until we've implemented adaptive concurrency and/or records-per-fetch. Signed-off-by: Dimitar Dimitrov <[email protected]> * Remove redundant panic Signed-off-by: Dimitar Dimitrov <[email protected]> --------- Signed-off-by: Dimitar Dimitrov <[email protected]> (cherry picked from commit 3ea4cf4) Co-authored-by: Dimitar Dimitrov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
Striking the right configuration for ongoing fetching depends a lot on the characteristics of cluster. franz-go is better at adaptive concurrency, so we fall back to it until we've implemented adaptive concurrency and/or records-per-fetch.
Which issue(s) this PR fixes or relates to
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.