objtools: fix 5GiB+ server-side copies on S3 #9357
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a rebased, fixed, and tested version of #8427. I'm opening a new PR since I let that one rot while it was waiting to be tested (sorry!).
What this PR does
The primary motivation of this change is allowing
undelete-blocks
to recover very large blocks in S3 where anindex
object may exceed 5GiB.A server-side copy on S3 with minio's
CopyObject
fails if the object is greater than 5GiB in size. When testing the previous PR I found a bug that was fixable by supplyingMatchRange: true
in thesrcOptions
forComposeObject
. Upon further inspection I realized the byte ranges don't need to be created at all becauseComposeObject
already handles it.I commented that I don't think this approach works cross-region. The minio code tries to double check the source object size rather than allowing it to be supplied, but it unfortunately uses the destination client while doing so. Our test buckets don't work in that particular scenario anyway for other reasons so I was unable to directly verify this.
Manually tested using
copyprefix
to copy a 6GiB object from one prefix to another. It fails before this change and succeeds with it. A sha256 of the content of the copied object matched the source.Which issue(s) this PR fixes or relates to
Fixes https://github.com/grafana/mimir-squad/issues/2163
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]
.about-versioning.md
updated with experimental features.