Backup

[altafan]

At the moment, when recovering a wallet from a seed, the only thing restored is basically it's status, meant as accounts, addresses, txsa dn utxos. Thanks to some env vars and to the very specific structure of the daemon's embedded wallet we're able to restore its markets in term of just associating base and quote asset to a specific wallet account. We lose all those info like their fees, strategies etc.

In order to facilitate the restoration of all daemon's markets, we plan to add a new rpc ExportBackupFile that basically creates a JSON object with all relative information about the daemon's wallet and markets.

The following JSON is an example of the content of such backup file and we should iterate over this, so feedbacks are more than welcome:

{
   "wallet": {
       "rootPath":  "m/84'/1776'",
       "accounts": [
           {
               "name": "...",
               "derivationPath": "...",
               "balance": { ... }
           },
           ...
       ]
   },
   "markets": [
       {
           "name": "...",
           "baseAsset": "...",
           "quoteAsset": "...",
           "strategy": "...",
           "fees": {
               "percentageFee": <value>,
               "fixedFee": { ... }
           },
       },
       ...
   ]
}

@tiero @sekulicd what do you think of this as a starting point?

I also have an open question:

What should be the behavior of this rpc? Should it create the JSON file somewhere like for example in the datadir? Or, instead, should it send the serialzied json object within the response message and let the client eventually create the file?

[tiero]

Or, instead, should it send the serialzied json object within the response message

Yep lets keep it stateless, and return the JSON object in a string or something

[altafan]

Some considerations to revamp the discussion:

• this RPC should also be added to the v0.9 daemon to facilitate the migration to v1 with a wallet restoration + backup file. It's a great occasion to model "old" domains coming from v0.9 into the "new" format used by the v1.
• the backup response is missing info about txs (trades, deposits and withdrawals).
• if we take into consideration that there might be a high number of txs to be included in this JSON, it could become problematic to serialize it within the response message due to msg size limitations. In light of this, it might be preferable to create a backup.json file into the datadir.

Improvements to the original proposal:

• the RPC should be protected with the password, named ExportBackup and exposed by the Operator service
• the wallet section should include msRootPath in case a ms ocean wallet is behind the scenes (v1 daemon)
• the account section should include xpubs, no need to export the master blinding key since it can be always derived from mnemonic (ss wallet account) or xpubs (ms wallet account)
• each market should include base and quote asset precisions, fees should be replaced by percentage and fixed fees, both with values specified on base and quote assets as per v1 spec.
• spec for transactions:

"trades": [
    {
        "id": "...",
        "type": "...",
        "marketName": "...",
        "marketBaseAsset": "...",
	"marketQuoteAsset": "...",
	"marketPrice": {
             "basePrice": "...",
             "quotePrice": "..."
        },
	"marketPercentageFee": {
             "baseAsset": <value>,
             "quoteAsset": <value>
        },
	"marketFixedFee": {
             "baseAsset": <value>,
             "quoteAsset": <value>
        },
	"feeAsset": "...",
	"feeAmount": <value>,
	"status": {
	     "code": <value>,
	     "failed": <value>
	},
	"psetBase64": "...",
	"txid": "...",
	"txHex": "...",
	"expiryTime": <value>,
	"settlementTime": <value>,
	"swapRequest": {
	     "id": "...",
             "message": "..."
             "timestamp": <value>
	},         *Swap
	"swapAccept": {
	     "id": "...",
             "message": "..."
             "timestamp": <value>
	},
	"swapComplete": {
	     "id": "...",
             "message": "..."
             "timestamp": <value>
	},
	"swapFail": {
	     "id": "...",
             "message": "..."
             "timestamp": <value>
	}
   },
   ...
],
"deposits": [
    {
        "txid": "...",
        "accountName": "...",
        "totAmountPerAsset": {
            "<asset>": <value>,
            ...
        },
        "timestamp": <value>
    },
    ...
],
"withdrawals": [
    {
        "txid": "...",
        "accountName": "...",
        "totAmountPerAsset": {
            "<asset>": <value>,
            ...
        },
        "timestamp": <value>
    },
    ...
]

Translation from old to new format:

• market
  • single percentage fee value must be mapped to "percentageFee": { "baseAsset": , "quoteAsset": }`.
  • market name is missing in old format and should be calculated as hash(baseAsset + quoteAsset)
• trade
  • trade type is missing in old format and should be mapped to UNKNOWN
  • trade status codes should be mapped in accordance to how they changed in internal/core/domain/constants.go
  • in order to produce correct report, fee asset and amount must not be empty. The first should always be the sent asset (asset r), while the second can be calculated based on amount r and trade fees. Based on the proportion (amountR + fixedFeeAmount) : 100 - percentageFee = x : percentageFee that allows us to find the percentage fee amount, the total fee amount should be calculated with the following formula: feeAmount := percentageFeeAmount + fixedFeeAmount = ((amountR + fixedFeeAmount) * percentageFee / 100 - percentageFee) + fixedFeeAmount.
  • raw swap messages (byte slices) should be serialized as hex strings
• deposits
  • the old format refers to utxos with the word deposit, while the new format refers to transactions, therefore translating the deposits into the new format basically means reconstructing the required tx infos from a set of utxos
• withdrawals
  • the old format refers to tx in a slightly different way compared to v1 but the translation is straight-forward.

[tiero]

All good, but my main issue with datadir is that we cannot build GUI tools to perform "export" & "import"

If we assume "i have my 0.x daemon on the same machine where I am upgrading" that should be handled with a migration routine in V1 that if recognize a datadir with old data, will perform migration automatically under the hood.

The JSON idea was more I have my daemon (either 0.x or 1.x) running and I want to spin-up an exact clone somewhere else in a different infrastructure. Maybe we should consider the bytes option instead of JSON? Ideally can be a server-stream to chunk multiple pieces together in order to be able to handle the scale of a huge file (with grpc-gateway will still be base64 strings I guess, but with chunking make it scalable)

[tiero]

So as wrap-up, let's split up the task in priorities:

1. v0.x to v1 migration routine (badger to badger I guess, as we will rollout postgres much later on)
2. Export server-side streaming RPCs to download
3. Investigate client-side streaming to upload (is this impossible with JSON HTTP I suppose?)