Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactored object store logging for DataFusion compaction #3974

Open
m09526 opened this issue Dec 18, 2024 · 0 comments
Open

Refactored object store logging for DataFusion compaction #3974

m09526 opened this issue Dec 18, 2024 · 0 comments
Assignees
Labels
compactions-module enhancement New feature or request

Comments

@m09526
Copy link
Member

m09526 commented Dec 18, 2024

Background

The DataFusion compactor includes a LoggingObjectStore wrapper around the ObjectStore trait. This is responsible for logging requests made to the underlying object store (S3 in our case).

Description

As a base for improving the use of ObjectStore implementations, we want to re-factor the existing implementation of LoggingObjectStore. This is to remove some pointer indirection, add unit tests, and reduce noise in the code. This will form the basis of future work to reduce the number of GET requests to S3 that the DataFusion compactor makes.

Analysis

Resolving this issue in itself should make the compaction code that deals with remote object stores cleaner.

@m09526 m09526 added enhancement New feature or request compactions-module pr-base-for-stacking Base for stacked pull requests (a dependency for others) labels Dec 18, 2024
@m09526 m09526 self-assigned this Dec 18, 2024
@patchwork01 patchwork01 removed the pr-base-for-stacking Base for stacked pull requests (a dependency for others) label Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compactions-module enhancement New feature or request
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

2 participants