Make getnetworksolps & getnetworkhashps RPCs more efficient by caching difficulties and times

[teor2345]

Motivation

In ticket #6688 we did some quick fixes to speed up the getnetworksolps & getnetworkhashps RPCs.

This ticket is for a long-term fix that is even faster, and supports quick work calculations over the full block range.

Specifications

For each block other than the genesis block , nTime MUST be strictly greater than the median-time-past of that block.
For each block at block height 2 or greater on Mainnet , or block height 653606 or greater on Testnet , nTime MUST be less than or equal to the median-time-past of that block plus 90 · 60 seconds.

https://zips.z.cash/protocol/protocol.pdf#blockheader

Read the state upgrade instructions before starting this ticket:
https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/state-db-upgrades.md#upgrading-the-state-database

Complex Code or Requirements

This ticket requires a state upgrade:

• store the cumulative work for each height in the finalized state (16 bytes per difficulty, should use a compact encoding)
• store the block times for each height in the finalized state (4 bytes per time)

Then changes to the RPC calculations:

• do the work calculations by subtracting the end height cumulative work from the start height cumulative work
• find the minimum and maximum times by reading all the times from the index (this will be fast enough because reading all the block headers only takes a few minutes)

complex optimisation - don't do this

If that's not fast enough, we can:

• read the first PoWMedianBlockSpan/2 blocks' times: this is always the minimum because the "strictly greater than the median-time-past" rule requires the time to increase when the median increases
• reading the last PoWMedianBlockSpan/2 * 4500 blocks' times: this is always the maximum because the "less than or equal to the median-time past plus 60*90 seconds" rule limits the maximum to 4500 greater than the median, and the "strictly greater than the median-time-past" rule guarantees regular time increases, so the median
• and height 0..=2 on mainnet or height 0..=653606 on testnet, if they are inside the range, because these rules don't apply to them,
• or reading the full range if it is smaller than these ranges

Then carefully test they produce the same results as the full range over a large number of random ranges. (Using a proptest?)

Testing

The original tests should pass before and after this change, but we should extend their ranges.

We should manually check that the RPC is faster on large ranges, using the RPC test cases from PR #7647.

Related Work

• getnetworksolps & getnetworkhashps RPCs hang with large num_blocks #6688