Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update thanos and add block_ids_fetcher to bucketindex #5681

Merged
merged 8 commits into from
Nov 28, 2023
Merged

update thanos and add block_ids_fetcher to bucketindex #5681

merged 8 commits into from
Nov 28, 2023

Conversation

wenxu1024
Copy link
Contributor

@wenxu1024 wenxu1024 commented Nov 27, 2023 •
edited
Loading

What this PR does:

  1. Update Thanos to latest version.
  2. Added bucket-index-ids-fetcher to allow compactor to use bucket-index to get the list of active and partial blocks. This could decrease the meta data syncer time from more than 10m to less than 1m for large tenants.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

yeya24
yeya24 reviewed Nov 28, 2023
View reviewed changes
Copy link
Contributor

@yeya24 yeya24 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, I like this pr. Let's make sure we have enough tests coverage for the new block IDs fetcher and the compactor itself.
Unfortunately we don't have integration tests for compactor rn. We should have that at some point.

pkg/compactor/bucket-index-block-ids-fetcher.go Outdated Show resolved Hide resolved
pkg/compactor/bucket-index-block-ids-fetcher.go Outdated Show resolved Hide resolved
pkg/compactor/bucket-index-block-ids-fetcher.go Outdated Show resolved Hide resolved
pkg/compactor/compactor.go Outdated Show resolved Hide resolved
pkg/compactor/bucket-index-block-ids-fetcher.go Outdated Show resolved Hide resolved
pkg/compactor/bucket-index-block-ids-fetcher.go Outdated Show resolved Hide resolved
pkg/compactor/bucket-index-block-ids-fetcher.go Outdated Show resolved Hide resolved
pkg/compactor/bucket-index-block-ids-fetcher.go Outdated
}
}

func (f *BucketIndexBlockIDsFetcher) GetActiveAndPartialBlockIDs(ctx context.Context, ch chan<- ulid.ULID) (partialBlocks map[ulid.ULID]bool, err error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make sure we have unit tests to cover this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added unit test.

@yeya24
Copy link
Contributor

yeya24 commented Nov 28, 2023

Btw do we still need #5641?

@wenxu1024
Copy link
Contributor Author

Btw do we still need #5641?

Thanks. We can close that one.

@wenxu1024 wenxu1024 changed the title update thanos and add bucket-index-ids-fetcher to compactor update thanos and add bucket-index-ids-fetcher to bucketindex Nov 28, 2023
@wenxu1024 wenxu1024 changed the title update thanos and add bucket-index-ids-fetcher to bucketindex update thanos and add block_ids_fetcher to bucketindex Nov 28, 2023
@wenxu1024
remove TODO
52c8a50 
Signed-off-by: Wen Xu <[email protected]>
yeya24
yeya24 approved these changes Nov 28, 2023
View reviewed changes
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 Thanks!

danielblando
danielblando approved these changes Nov 28, 2023
View reviewed changes
Copy link
Contributor

@danielblando danielblando left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice change

@yeya24 yeya24 merged commit 7f8d194 into cortexproject:master Nov 28, 2023
14 checks passed
@wenxu1024 wenxu1024 mentioned this pull request Dec 19, 2023
Merged
3 tasks
@yeya24 yeya24 mentioned this pull request Mar 25, 2024
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yeya24 yeya24 yeya24 approved these changes

@danielblando danielblando danielblando approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wenxu1024 @yeya24 @danielblando