Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add APIs necessary for log compaction algorithms #217

Merged
merged 8 commits into from
Apr 20, 2022
Merged

add APIs necessary for log compaction algorithms #217

merged 8 commits into from
Apr 20, 2022

Conversation

staltz
Copy link
Member

@staltz staltz commented Apr 20, 2022

Context: #199

Problem 1: we need to know when is indexing ongoing and/or when is querying ongoing, because we don't want compact to happen in parallel with those.

Problem 2: if compaction is ongoing, we can't allow JITDB queries to start.

Problem 3: reindex has to rebuild also the core indexes.

Solution:

  • Introduces the public APIs indexingActive and queriesActive which are Obz's that emit numbers of how many are active at the moment.
  • Monitor the status of compaction and guard the queries to wait for compaction to end first
  • reindex rebuilds also the core indexes

These changes are already used and tested in ssbc/ssb-db2#339

@staltz staltz requested a review from arj03 April 20, 2022 12:06
@github-actions
Copy link

Benchmark results

Part Speed Heap Change Samples
Count 1 big index (3rd run) 0.38ms ± 0.03ms -1.8 kB ± 10.57 kB 59
Create an index twice concurrently 500.15ms ± 2.67ms 18.21 kB ± 25.39 kB 109
Load core indexes 0.8ms ± 0.01ms 92.7 B ± 181.62 B 9685
Load two indexes concurrently 407.67ms ± 2.56ms -83.75 kB ± 106.33 kB 24
Paginate 10 results 26.35ms ± 1.64ms 11.51 kB ± 16.76 kB 29
Paginate 20000 msgs with pageSize=5 6103.73ms ± 86.69ms 145.33 kB ± 973.87 kB 5
Paginate 20000 msgs with pageSize=500 480.78ms ± 2.46ms 231.13 kB ± 415.86 kB 23
Query 1 big index (1st run) 742.65ms ± 11.93ms -2.83 kB ± 38.49 kB 74
Query 1 big index (2nd run) 251.61ms ± 0.62ms 1.71 kB ± 10.18 kB 59
Query 3 indexes (1st run) 1068.12ms ± 14.93ms -52.37 kB ± 150.24 kB 50
Query 3 indexes (2nd run) 218.64ms ± 1.08ms 21.41 kB ± 24.41 kB 62
Query a prefix map (1st run) 225.85ms ± 1.94ms 28.97 kB ± 534.77 kB 26
Query a prefix map (2nd run) 14ms ± 0.7ms -12.99 kB ± 45.61 kB 26

if (coreIndexNames.includes(indexName)) return cb()

if (index.lazy) {
if (coreIndexNames.includes(indexName)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, reindex can now be done when things are removed, so this changes the core indexes. Before when it was only encrypted, these core indexes does not change.

Copy link
Member

@arj03 arj03 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was really easy to review. Nice work.

@staltz staltz merged commit 2d40781 into master Apr 20, 2022
@staltz staltz deleted the compact branch April 20, 2022 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants