Limit number of parallel bitcoin rpc connections

[sander2]

Related to interlay/interlay.github.io#275 .

At first I didn't understand how spancoin could run into the bitcoin connection limit of 16 when he was running only 7 vaults, as we use a blocking bitcoin api. However, I did some testing, and it shows that we indeed make several requests in parallel:

Oct 19 09:54:25.889 DEBUG bitcoincore_rpc: JSON-RPC request: getblockhash [2377248]    
Oct 19 09:54:25.967 DEBUG bitcoincore_rpc: JSON-RPC request: listunspent []    
Oct 19 09:54:25.967 DEBUG bitcoincore_rpc: JSON-RPC request: getblockhash [2377247]    
Oct 19 09:54:26.002 DEBUG bitcoincore_rpc: JSON-RPC error for getblockhash: RpcError { code: -8, message: "Block height out of range", data: None }    
Oct 19 09:54:26.080 TRACE bitcoincore_rpc: JSON-RPC response for getblockhash: "0000000045a081ced69de01bb65381e0c0e432cb43020bb629e35191e3638df0"    
Oct 19 09:54:26.080 TRACE bitcoincore_rpc: JSON-RPC response for listunspent: []

The reason that this happens is that while each task blocks on the bitcoin rpc, tasks can be run on different threads, resulting in this behavior. Maybe we should look into limiting the number of parallel connections, although I am a little bit afraid of unforeseen consequences such as timeouts in subxt..

[spazcoin]

I think it's totally ok for multiple threads to be blocking waiting for RPC responses. In fact, I'd encourage increasing the timeout so that they're willing to wait a little longer. Increasing the number of bitcoin RPC workers seems like a more appropriate scaling strategy. So I'd suggest:

• Increase thread blocking timeout.
• If a vault thread times out, print out a message suggesting the bitcoin.conf "rpcworkqueue" value be increased and point to the docs