Self-hosted RPC nodes with fast confirmations, health and performance monitoring, and service guarantees

[polymorpher]

At this time, all transactions are performed by relayer through relayer accounts, which routes all transactions to the blockchain via public RPC nodes (i.e. api.harmony.one and api.s0.hmny.io). Over the past few months, users often experience connection errors and transaction failures when there is a high number of users using the wallet concurrently. See preliminary solutions in #259 which attempted to address this issue.

However, disruption of services still occur from time to time. In some instances, transactions are silently dropped in the RPC server without any response to the relayer. Subsequent transactions also become stuck in the relayer.

To address this issue in a fundamental way, dedicated validators nodes that also serve RPC requests are needed. The dedicated node shall process only transactions sent by selected relayers. This can be achieved through special firewall settings. For development and security purposes, the nodes also need to run custom-compiled version of the underlying blockchain (e.g. http://github.com/harmony-one/harmony). Some RPC endpoints should also be repeatedly called to monitor the health of the node, and whether it caught up with the blockchain, for example, via blockHeader api https://docs.harmony.one/home/developers/api/methods/blockchain-related-methods/hmy_latestheader . The overall performance can be monitored through relayer monitoring API (see #290), and in the future, in a more detailed way through granular analytics #296

[polymorpher]

This is completed. Private nodes are deployed and the validator can be seen at https://staking.harmony.one/validators/mainnet/one1j5gzpfcywjpapytn4awhrd7feas0yulk98mnyp