Allow readers to read consistently #1043
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Oct 17, 2024
This was referenced Oct 17, 2024
blt
force-pushed
the
blt/ensure_file_size_updates_due_to_getattr
branch
from
8f3227c
to
3cfa40a
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
06b8ee6
to
d165859
Compare
blt
force-pushed
the
blt/ensure_file_size_updates_due_to_getattr
branch
from
3cfa40a
to
61386a0
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
d165859
to
7e4aed0
Compare
blt
force-pushed
the
blt/ensure_file_size_updates_due_to_getattr
branch
from
61386a0
to
8f3f944
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
7e4aed0
to
54cf3ea
Compare
blt
force-pushed
the
blt/ensure_file_size_updates_due_to_getattr
branch
from
8f3f944
to
854d58d
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
54cf3ea
to
ea176e5
Compare
blt
force-pushed
the
blt/ensure_file_size_updates_due_to_getattr
branch
from
854d58d
to
340087f
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
ea176e5
to
207dd72
Compare
blt
force-pushed
the
blt/ensure_file_size_updates_due_to_getattr
branch
from
340087f
to
1d0b1f0
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
207dd72
to
a03cff4
Compare
blt
force-pushed
the
blt/ensure_file_size_updates_due_to_getattr
branch
from
1d0b1f0
to
8c02a22
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
a03cff4
to
5424a84
Compare
This was referenced Oct 25, 2024
Merged
blt
force-pushed
the
blt/ensure_file_size_updates_due_to_getattr
branch
from
191bcb0
to
55b85e2
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
19e5b4f
to
d62fce9
Compare
goxberry
reviewed
Oct 28, 2024
Comment on lines
+417
to
+441
| while remaining > 0 { | ||
| // The plan is this. We treat the blocks as one infinite cycle. We | ||
| // map our offset into the domain of the blocks, then seek forward | ||
| // until we find the block we need to start reading from. Then we | ||
| // read into `data`. | ||
|
|
||
| let offset_within_cycle = current_offset % total_cycle_size; | ||
| let mut block_start = 0; | ||
| for block in blocks { | ||
| let block_size = block.total_bytes.get() as usize; | ||
| if offset_within_cycle < block_start + block_size { | ||
| // Offset is within this block. Begin reading into `data`. | ||
| let block_offset = offset_within_cycle - block_start; | ||
| let bytes_in_block = (block_size - block_offset).min(remaining); | ||
|
|
||
| data.extend_from_slice( | ||
| &block.bytes[block_offset..block_offset + bytes_in_block], | ||
| ); | ||
|
|
||
| remaining -= bytes_in_block; | ||
| current_offset += bytes_in_block; | ||
| break; | ||
| } | ||
| block_start += block_size; | ||
| } |
There was a problem hiding this comment.
Had to stare at this loop for a while to appreciate the subtleties involved (reads crossing block boundaries, cycle boundaries). Neat stuff.
goxberry
approved these changes
Oct 28, 2024
blt
force-pushed
the
blt/ensure_file_size_updates_due_to_getattr
branch
2 times, most recently
from
b5e67d9
to
4206504
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
2 times, most recently
from
0e316da
to
4fa0e24
Compare
blt
changed the base branch from
blt/ensure_file_size_updates_due_to_getattr
to
graphite-base/1043
blt
force-pushed
the
graphite-base/1043
branch
from
4206504
to
34e3967
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
4fa0e24
to
68a0760
Compare
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
68a0760
to
5501775
Compare
This commit exposes a `read_at` on the block cache allowing callers to read the same data from offset X over and over again. This means that when we expose data from a block cache in the logrotate filesystem we are able to correctly do `wc -l` and similar. Signed-off-by: Brian L. Troutwine <[email protected]>
Signed-off-by: Brian L. Troutwine <[email protected]>
blt
force-pushed
the
blt/allow_readers_to_read_consistently
branch
from
5501775
to
ffae4d1
Compare
Merge activity
|
This was referenced Oct 28, 2024
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This commit exposes a
read_aton the block cache allowing callers toread the same data from offset X over and over again. This means that
when we expose data from a block cache in the logrotate filesystem we
are able to correctly do
wc -land similar.