disable live in level indexes while reindexing is ongoing

[staltz]

Context

When running deleteFeeds-then-compact-then-reindex experiments in Manyverse, I noticed that the Activity tab highlighted as "new", which didn't make sense because there was no new content added (replication was turned off).

This happened consistently, regardless of what dataset I was using (production or ssb-fixtures).

See also #306 (comment)

Problem

It turned out the problem was with leveldb live. The way that pull-level and level-post work is by adding listeners to the leveldb EventEmitter 'put', 'del', 'batch': https://github.com/dominictarr/level-post/blob/9021f153f159aa1d13233908ebe616f650e78f22/index.js#L44-L46

Obviously, during reindexing, we use batch to recreate the index, and this triggers the "live" side of pull-level, which then triggers fullMentions live stream. The Activity tab in Manyverse uses fullMentions. But I also think that any other leveldb index which uses live, such as aboutSelf, would be affected in a similar way.

Solution

Not the cleanest solution, but I wanted to find a solution that is general for all our leveldb indexes AND contained to ssb-db2 repo to not require each index to be updated (which is error prone).

I figured that we can just rip out all the EventEmitter listeners when the leveldb index is "reset" (which so far ONLY happens after compaction), and then wait until the index catches up with the full log, and then put those listeners back in.

1st ❌ 2nd ✔️

[github-actions]

Benchmark results

Part	Duration
add 1000 elements	494ms
add 1000 private box1 elements	1155ms
unbox 1000 private box1 elements first run	125ms
unbox 1000 private box1 elements second run	104ms
add 1000 private box1 elements	1141ms
query 1000 elements first run	28ms
query 1000 elements second run	13ms
add 1000 private box2 elements	634ms
unbox 1000 private box2 elements first run	361ms
unbox 1000 private box2 elements second run	355ms
Migrate (+db1)	11704ms
Migrate (alone)	4097ms
Migrate (+db1 +db2)	8059ms
Migrate (+db2)	6684ms
Migrate continuation (+db2)	930ms
Memory usage without indexes	739.18 MB = 32.26 MB + etc
Initial indexing	693ms
Initial indexing maxCpu=86	4572ms
Initial indexing compat	900ms
Two indexes updating concurrently	891ms
key one initial	44ms
key two	0ms
key one again	1ms
reboot and key one again	45ms
latest root posts	633ms
latest posts	9ms
votes one initial	552ms
votes again	0ms
hasRoot	395ms
hasRoot again	0ms
author one posts	296ms
author two posts	18ms
dedicated author one posts	330ms
dedicated author one posts again	1ms
Maximum memory usage	880.72 MB = 55.14 MB + etc
Indexes folder size	9.97mb

[arj03]

I think the solution of adding it here is fine.

Wow performance must have be pretty bad before because of all of these live events hammering things.

[github-actions]

Benchmark results

Part	Duration
add 1000 elements	485ms
add 1000 private box1 elements	1028ms
unbox 1000 private box1 elements first run	104ms
unbox 1000 private box1 elements second run	100ms
add 1000 private box1 elements	1117ms
query 1000 elements first run	37ms
query 1000 elements second run	12ms
add 1000 private box2 elements	583ms
unbox 1000 private box2 elements first run	338ms
unbox 1000 private box2 elements second run	329ms
Migrate (+db1)	11801ms
Migrate (alone)	4253ms
Migrate (+db1 +db2)	8128ms
Migrate (+db2)	6415ms
Migrate continuation (+db2)	877ms
Memory usage without indexes	733.06 MB = 36.08 MB + etc
Initial indexing	673ms
Initial indexing maxCpu=86	4179ms
Initial indexing compat	831ms
Two indexes updating concurrently	990ms
key one initial	46ms
key two	0ms
key one again	1ms
reboot and key one again	48ms
latest root posts	556ms
latest posts	3ms
votes one initial	471ms
votes again	1ms
hasRoot	361ms
hasRoot again	0ms
author one posts	263ms
author two posts	14ms
dedicated author one posts	310ms
dedicated author one posts again	0ms
Maximum memory usage	1002.48 MB = 59.81 MB + etc
Indexes folder size	9.97mb