Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor fuse pruner #13853

Closed
SkyFan2002 opened this issue Nov 29, 2023 · 3 comments
Closed

Refactor fuse pruner #13853

SkyFan2002 opened this issue Nov 29, 2023 · 3 comments
Assignees

Comments

@SkyFan2002
Copy link
Member

Summary
Currently pruning process is blocking the entire pipeline. By integrating the pruning process into the pipeline, the parallelism of pruning and query execution can be enhanced.

Related PR: #11552

@Xuanwo
Copy link
Member

Xuanwo commented Oct 29, 2024

Hi, when refactoring the fuse pruner, let's keep in mind that Databend has multiple engines like Hive Table, Delta Table, and Iceberg Table. It would be better to refactor in a way that can be shared among them.

cc @SkyFan2002 @dqhl76 @sundy-li

@dqhl76
Copy link
Collaborator

dqhl76 commented Oct 29, 2024

It would be better to refactor in a way that can be shared among them.

I am planing to split the different table engines' pruning process into processors. Some of them have probablity to be refactored as a sharing processor. Plan to first start with fuse engine. Thanks for the reminding, will keep in mind.

@dqhl76
Copy link
Collaborator

dqhl76 commented Nov 19, 2024

Duplicate with #16626. Tracking in that.

@dqhl76 dqhl76 closed this as completed Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants