Synchronous race conditions with ssb-friends

[staltz]

Context and problem

See #55

ssb-ebt / ssb-friends have always been in a fragile state where they barely work, and if you change something to be async (like via a Promise or just with a 1ms setTimeout), then they stop working.

Suppose there are three peers, Alice, Bob, and Carol. The network connections are set up as Alice <==> Carol <==> Bob, so that Carol is the intermediate.

1. Alice publishes an new message that she is blocking Bob
2. Carol receives that message via ssb-ebt
3. Carol should process that message in ssb-friends and register in Carol's EBT that Alice blocks Bob
4. Carol should update connected peers (i.e. Bob) about the new message, unless the connected peer should not receive it (because of blocks)

We need (3) to happen strictly before (4), and currently ssb-ebt is working fine, but this is a fragile situation, and any addition of asynchronous work in ssb-ebt or ssb-friends could potentially break this, causing (4) to happen before (3), and then causing Bob to receive Alice's msg, when he shouldn't.

Proposed solutions

We need to create explicit dependencies between these two computations, but it's a bit challenging because it's spread across many modules (ssb-db2, ssb-ebt, epidemic-broadcast-trees, ssb-friends, layered-graph, ssb-replication-scheduler).

Here's one possible solution:

• When ssb-friends db index processes a new message, it also passes some metadata on that message down to layered-graph
• layered-graph does its Dijkstra calculations and when it updates event listeners, it attaches the metadata it received
  • This module doesn't actually need to know anything about SSB or messages, it just needs to pass along the metadata that it received
• ssb-friends public APIs as listeners of layered-graph would pick up the attached metadata and pass it on via hopStream and graphStream
• ssb-ebt's usage of sbot.post to track the database would be moved to ssb-replication-scheduler
• ssb-replication-scheduler can now keep track of which db messages caused which hopStream updates, and by controlling sbot.post it can make sure that ssb-ebt is updated with the new message only after it has been processed by hopStream (which calls sbot.ebt.block())

Even with this proposed solution, we have to be careful that every ongoing sbot.ebt.block() (in case it has asynchronous tasks, like promises or whatnot) is completed before ebt.onAppend() is called.