You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Development of a Pruning Pipeline: A new 'pruning pipeline' system has been added to help manage data more efficiently. This system includes the option for users to decide if they want to enable or disable it, and a way to read the setting to understand if it's enabled or not.
Refining of Data Reading Method: The method to read data has been adapted to better handle initialization of segments and includes the new pruning logic.
Enhancements of Pruning Mechanism: The comprehensive pruning mechanism now also includes caching strategies and more sophisticated ways of handling segments.
Additional Pruning Operations: More pruning operations have been added to allow asynchronous segment pruning, improving execution speed.
Boosted Synchronization Capabilities: A new external library (or "crate") named parking_lot has been included to enhance data synchronization capabilities.
Asynchronous and Synchronous Pruning Features: The PR includes new asynchronous and synchronous block pruning features, improving execution speed and flexibility.
Metadata Management Techniques: Techniques for handling and representing block metadata have been developed. This includes probability-based sampling of metadata, extraction of metadata from segments, and creation of metadata pairs to represent pruning results.
Note: The PR also contains several newly-added files, and overall, the new developments contribute to more efficient data handling and improved performance. Several methods were made public to improve accessibility, and minor codes were refactored to enhance clarity and modularity.
pr-refactorthis PR changes the code base without new features or bugfix
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
bot
added
the
pr-refactor
dqhl76
changed the title
zhang2014
added
ci-benchmark
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
part of #16626
Split the block prune part and put into processor
Tests
Type of change
This change is