Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use sizeDiff=1 to represent holes filled #71

Merged
merged 1 commit into from
Apr 24, 2022
Merged

use sizeDiff=1 to represent holes filled #71

merged 1 commit into from
Apr 24, 2022

Conversation

staltz
Copy link
Member

@staltz staltz commented Apr 20, 2022

Context: ssbc/jitdb#199

Problem

In my ssb-db2 PR ssbc/ssb-db2#339, I had a if (sizeDiff > 0) resetIndexes() but this misses a corner case. sizeDiff is only greater than zero if truncation removed at least one block. But sizeDiff === 0 can mean two very different things:

  • Compaction found no holes at all, and the log does not need compaction, it's left intact
  • Compaction found holes but only a few, and performed "shifting" of records to rewrite the log, but the total size remained the same

We need to rebuild indexes in the 2nd case, but not in the 1st. So we need a way of knowing the difference between these cases.

Solution

Use sizeDiff === 1 as a special case where no blocks have been truncated (the log size remained the same) but some holes have been filled, thus requiring ssb-db2 to rebuild indexes.

I think it's safe to say that sizeDiff is 1 byte because the minimum size of a block is 3 bytes because just the "End Of Block" marker takes 2 bytes.

@staltz staltz requested a review from arj03 April 20, 2022 16:10
unshiftedBlockIndex = Math.floor(
state.unshiftedOffset / log.blockSize
)
unshiftedBlockIndex = Math.floor(unshiftedOffset / log.blockSize)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These 3 changes are unrelated to the PR problem, they are just nice refactors

Copy link
Member

@arj03 arj03 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep

@staltz staltz merged commit 1c07825 into master Apr 24, 2022
@staltz staltz deleted the holes-filled branch April 24, 2022 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants