check eth provider status

[kilrau]

Problem

I am connecting to my own local geth (mainnet in this example):

2020-06-17T09:43:17.602Z [ConnextConnect] Called connect with {
  "nodeUrl": "https://connext.odex.dev/api",
  "ethProviderUrl": "http://192.168.123.44:8545"
}, and signer using private key
[1592386997665] INFO  (70 on connext): Mnemonic set successfully
2020-06-17T09:43:18.072Z [CFCore] CFCore signer address: 0xE7Ff258482EaEaFB3aEAc68FF1C7231A723D0bcC
2020-06-17T09:43:18.089Z [CF-Sync] Executing with params: {"multisigAddress":"0x78fB..50A8"}
2020-06-17T09:43:18.133Z [NodeApiClient] Node responded to indra5GYKLaTiQg6SQh8iztaAkYkZFowfbDMw8LSnZdwpv2YdjWxq5h.lock.acquire.0x78fB1883DC1a172c67C2f85C464Cb38A0fc350A8 request in 43 ms

Problem: the client doesn't seem to check eth.syncing status of the eth provider on connect since in the moment above I actually just restarted geth and it was still catching up with blocks/states:

Welcome to the Geth JavaScript console!

instance: Geth/v1.9.15-stable-0f77f34b/linux-amd64/go1.14.4
at block: 10282488 (Wed Jun 17 2020 09:42:15 GMT+0000 (UTC))
 datadir: /root/.ethereum
 modules: admin:1.0 debug:1.0 eth:1.0 ethash:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

> eth.syncing
{
  currentBlock: 10282489,
  highestBlock: 10282501,
  knownStates: 499940845,
  pulledStates: 499940845,
  startingBlock: 10282310
}

What I think should be done:

• have a dedicated http error code for the eth provider being out-of-sync
• client checks syncing status of eth provider on connect and reports "Waiting for chain backend to finish syncing. Retrying in 5s...` or something along these lines and then retries.
• during operation, regularly (e.g. 10s) hit an endpoint (e.g. eth.syncing) at the eth-provider to verify its connected and well. Report if errors come up and retry connecting. Since with eth its common to connect to a remote eth provider, I think we should have this as a p2p ping/pong heart beat. We hit several invalid response - 0 erros in our swap tests which seem to hint at a problem with the eth provider, but the client was never reporting any issues... and showed Ready in xud.

[ghost]

I believe we discussed offline with @ArjunBhuptani and didn't find a good solution since the goal is for the connext client to be provider agnostic. Let's figure out a solution for this as I expect more consumers of the client will face this issue.

[kilrau]

I agree on being provider agnostic. To achieve that, the client could have a flag --ignoreethproviderhealth to treat the eth provider health status purely informational.

That makes the health check geth-specific and since moth eth providers are geth-based will work and additionally has an option to connect to other eth providers by setting above flag which disabled the health check.

[ArjunBhuptani]

We definitely want to do this, but realistically it wont be a priority for some time (since users can check provider health themselves in the process of passing the provider in). Right now the biggest focus on our end is finding and eliminating any remaining channel-breaking bugs/edge cases that may exist (e.g. at high throughput or poor network latency).

Just want to be totally upfront -- it's absolutely something we think is important mid term.

[kilrau]

since users can check provider health themselves in the process of passing the provider in

Sure, but that's one-time only. An eth provider can get unhealthy after a while and in that case the connext client should stop processing payments since it couldn't go on-chain to dispute them. So it's somewhat security related imho.

[kilrau]

How about using something like https://www.anydot.dev/ as a fallback for the connext client going on-chain and it's own eth provider being unavailable or maybe as preferred way even? I am sure @stonecoldpat would like that.

[kilrau]

have a dedicated http error code for the eth provider being out-of-sync

Maybe we can start with this :)