Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(stream): add kafka backfill executor #14172

Merged
merged 30 commits into from
Mar 11, 2024
Merged

feat(stream): add kafka backfill executor #14172

merged 30 commits into from
Mar 11, 2024
+1,311 −110

Conversation

xxchan
Copy link
Member

@xxchan xxchan commented Dec 24, 2023 •
edited
Loading

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

risingwavelabs/rfcs#72

This PR adds the executor part.
There're still some stylish issue, and also need to add metrics & benchmark. But I hope to get the core logic reviewed and gain some feedback first.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added test labels as necessary. See details.
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

@xxchan xxchan mentioned this pull request Dec 24, 2023
Merged
9 tasks
@xxchan
Copy link
Member Author

xxchan commented Dec 24, 2023 •
edited
Loading

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @xxchan and the rest of your teammates on Graphite Graphite

@xxchan xxchan mentioned this pull request Dec 24, 2023
Closed
9 tasks
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch from a7da39f to 5cdfc52 Compare December 24, 2023 06:29
@xxchan xxchan force-pushed the 12-23-feat_meta_align_splits_with_source_actor_with_an_upstream branch from 96e7ea8 to 077b948 Compare December 24, 2023 06:46
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch 3 times, most recently from ce0ef32 to 7c7c890 Compare December 24, 2023 09:50
@xxchan xxchan force-pushed the 12-23-feat_meta_align_splits_with_source_actor_with_an_upstream branch from 077b948 to 35c26f2 Compare December 26, 2023 05:10
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch from 7c7c890 to f012459 Compare December 26, 2023 05:10
@xxchan xxchan force-pushed the 12-23-feat_meta_align_splits_with_source_actor_with_an_upstream branch from 35c26f2 to 959df1c Compare December 26, 2023 15:09
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch from f012459 to a44b9dd Compare December 26, 2023 15:09
@xxchan xxchan force-pushed the 12-23-feat_meta_align_splits_with_source_actor_with_an_upstream branch from 959df1c to cf160ee Compare January 1, 2024 14:51
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch 2 times, most recently from b4d942b to 56ee844 Compare January 2, 2024 04:19
@xxchan xxchan force-pushed the 12-23-feat_meta_align_splits_with_source_actor_with_an_upstream branch from cf160ee to 2afd13e Compare January 9, 2024 17:31
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch from 56ee844 to 1e34118 Compare January 9, 2024 17:31
@xxchan xxchan mentioned this pull request Jan 9, 2024
Closed
12 tasks
@xxchan xxchan force-pushed the 12-23-feat_meta_align_splits_with_source_actor_with_an_upstream branch from 2afd13e to d3542d7 Compare January 11, 2024 08:11
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch from 1e34118 to 76d7b2a Compare January 11, 2024 08:11
@xxchan xxchan mentioned this pull request Jan 11, 2024
Merged
9 tasks
@xxchan xxchan force-pushed the 12-23-feat_meta_align_splits_with_source_actor_with_an_upstream branch from d3542d7 to 4e9fcfb Compare January 11, 2024 09:11
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch from 76d7b2a to eb8e2d9 Compare January 11, 2024 09:11
@xxchan xxchan force-pushed the 12-23-feat_meta_align_splits_with_source_actor_with_an_upstream branch from 4e9fcfb to 4edd11c Compare January 11, 2024 09:18
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch from eb8e2d9 to 435d4a2 Compare January 11, 2024 09:18
@xxchan xxchan force-pushed the 12-23-feat_meta_align_splits_with_source_actor_with_an_upstream branch from 4edd11c to abdf44c Compare January 12, 2024 02:02
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch from 435d4a2 to 979f250 Compare January 12, 2024 02:02
@xxchan xxchan force-pushed the 12-23-feat_meta_align_splits_with_source_actor_with_an_upstream branch from abdf44c to 8b7b439 Compare January 15, 2024 01:51
@xxchan xxchan force-pushed the 12-23-feat_stream_add_kafka_backill_executor branch from 979f250 to b7c3466 Compare January 15, 2024 01:51
fuyufjh
fuyufjh approved these changes Mar 7, 2024
View reviewed changes
Copy link
Member

@fuyufjh fuyufjh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM!

proto/stream_plan.proto Outdated Show resolved Hide resolved
src/stream/src/executor/source/kafka_backfill_executor.rs Outdated Show resolved Hide resolved
src/stream/src/executor/source/kafka_backfill_executor.rs Outdated Show resolved Hide resolved
src/stream/src/executor/source/kafka_backfill_executor.rs Outdated Show resolved Hide resolved
@xxchan xxchan mentioned this pull request Mar 7, 2024
Merged
9 tasks
BugenZhao
BugenZhao approved these changes Mar 8, 2024
View reviewed changes
Copy link
Member

@BugenZhao BugenZhao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM. It will take time to test to be more stable.

@xxchan xxchan mentioned this pull request Mar 11, 2024
Closed
@xxchan
Copy link
Member Author

xxchan commented Mar 11, 2024

Generally LGTM. It will take time to test to be more stable.

Yes. Let's test more before merging frontend code and/or before open the feature switch by default

@xxchan xxchan enabled auto-merge March 11, 2024 05:37
@xxchan xxchan changed the title feat(stream): add kafka backill executor feat(stream): add kafka backfill executor Mar 11, 2024
@xxchan xxchan added this pull request to the merge queue Mar 11, 2024
Merged via the queue into main with commit c7d5d9a Mar 11, 2024
31 of 32 checks passed
@xxchan xxchan deleted the 12-23-feat_stream_add_kafka_backill_executor branch March 11, 2024 07:34
@BugenZhao BugenZhao mentioned this pull request Mar 12, 2024
Closed
@xxchan xxchan mentioned this pull request Mar 13, 2024
Merged
11 tasks
@xxchan xxchan mentioned this pull request Mar 31, 2024
Closed
21 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@st1page st1page st1page left review comments

@BugenZhao BugenZhao BugenZhao approved these changes

@fuyufjh fuyufjh fuyufjh approved these changes

@StrikeW StrikeW Awaiting requested review from StrikeW

@chenzl25 chenzl25 Awaiting requested review from chenzl25

@tabVersion tabVersion Awaiting requested review from tabVersion

@yezizp2012 yezizp2012 Awaiting requested review from yezizp2012

Assignees
No one assigned
Labels
type/feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@xxchan @chenzl25 @fuyufjh @tabVersion @yezizp2012 @BugenZhao @st1page