Distributed in-memory key/value store for mapreduce #181
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
also improve sonic error handling. there is no need for handle to return a sonic::Result, it's better that the specific message has a Result<...> as their response as this can then be properly handled on the caller side
… and was therefore a bit misleading. remove it and add a send_with_timeout_retry method to normal connection with sane defaults in .send method
…ky Response::Set(Ok(())) for internal raft entries
in raft, writes are written to a majority quorom. if we have a cluster of 3 nodes, this means that we can only be sure that 2 of the nodes get's the data. the test might therefore fail if we are unlucky and check the node that didn't get the data yet. by having a cluster of 2 nodes instead, we can be sure that both nodes always receives all writes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the first step to move the mapreduce code to an AMPC model which will allow us to implement some distributed graph algorithms more efficiently. The main difference between the MPC and AMPC model is that we use a distributed key/value store for synchronization and to store the result for each round. This allows the algorithms to dynamically choose which segments of the previous rounds result it needs, and can therefore reduce the number of rounds needed overall.
The key/value store is a distributed
BTreeMap<Vec<u8>, Vec<u8>>where the key is routed to the correct shard based onmd5(key) % num_shards. Each shard uses raft based consensus for high availability. Shard rebalancing is currently not implemented, so adding/removing a shard will result in keys being routed to incorrect shards.In the paper they use remote direct memory access which we don't use here, but we should be able to get close to the same average latency by batching requests from the algorithm implementation which will hopefully amortize the overhead enough.