Replications

[ehsan6sha]

Replications mean the copies of a file that exist in a pool. Replication factor defines how many replications of a file can exist and all stores then are rewarded for storing a replica of the file (cids of the file). Replication can happen in a pool only. so when we submit a request for replication., we also identify which pool we want it to happen. Which means only members of that specific pool can replicate the file.

We have an API on blockchain that accepts a replication request for a list of cids: https://docs.fx.land/blockchain/Recipes/Manifest#batch-upload-manifest

When a file is uploaded (Push) the API is also called to submit a replication request by the client (on the mobile).
We also have an API that lists the replication requests: https://docs.fx.land/blockchain/Recipes/Manifest#get-available

The bloxes of a pool can call that API to see which cids are available to be replicated.
Since mobile devices are not stable (the Android or iOS can kill the process at anytime when the app is closed), we need to make sure the cid is uploaded to "a" blox when upload happens so that other bloxes can fetch it from that blox. This blox can be the owner's blox. But what if the owner blox is offline or user do not own a blox?
When a blox stores a cid (Pulls), it also announces it on dht, and other bloxes in the pool can use dht.FindProviders(cid) to see what bloxes are providing the cid. then they can connect to that blox and pull the cid.

currently, on the mobile device (where the upload happens) we mark the cids as "recent" when an upload happens and then submit the recent cids to the blockchain to be listed as available for replication. and Then we clear the list of recent cids if the API call is a success. This is to ensure we know which cids are added after each file upload

[ehsan6sha]

Currently, the problem is:
1- The only mechanism we have to know the replication requests is to periodically call the API which is not optimized.
2- Pull requests by 3rd parties must be authorized (so we need a handshake mechanism)
3- if 10 bloxes pull a cid that only 4 replications is needed for, then we do not have a mechanism to stop them. We need a mechanism that provider counts the number of pull requests and ensure it does not go above the replicaiton factor. For example if replication factor is 4, and it received 4 pull requests, it can send a wait signal to the 5th request and hold it for 1 minute. Then check the available replications by calling API and if it is still above 0 (meaning some of hte previous pullers did not submit the storage to blockchain) allows the extra pulls

[ehsan6sha]

I added a SearchValue and PutValue to put the number of replications in dht table when a replication is stored by a peer. however, for some reason it is not updated. I put value 1 for example from n3 and the peer 4 SearchValue gives 0 instead of 1. It is in the tests of blox "Example_storeManifest"