all: nuke total difficulty

[karalabe]

The total difficulty is the sum of all block difficulties from genesis to a certain block. This value was used in PoW for deciding which chain is heavier, and thus which chain to select. Since PoS has a different fork selection algorithm, all blocks since the merge have a difficulty of 0, and all total difficulties are the same for the past 2 years.

Whilst the TDs are mostly useless nowadays, there was never really a reason to mess around removing them since they are so tiny. This reasoning changes when we go down the path of pruned chain history. In order to reconstruct any TD, we must retrieve all the headers from chain head to genesis and then iterate all the difficulties to compute the TD.

In a world where we completely prune past chain segments (bodies, receipts, headers), it is not possible to reconstruct the TD at all. In a world where we still keep chain headers and prune only the rest, reconstructing it possible as long as we process (or download) the chain forward from genesis, but trying to snap sync the head first and backfill later hits the same issue, the TD becomes impossible to calculate until genesis is backfilled.

All in all, the TD is a messy out-of-state, out-of-consensus computed field that is overall useless nowadays, but code relying on it forces the client into certain modes of operation and prevents other modes or other optimizations. This PR completely nukes out the TD from the node. It doesn't compute it, it doesn't operate on it, it's as if it didn't even exist.

Caveats:

• Whenever we have APIs that return TD (devp2p handshake, tracer, etc.) we return a TD of 0.
• For era files, we recompute the TD during export time (fairly quick) to retain the format content.
• It is not possible to "verify" the merge point (i.e. with TD gone, TTD is useless). Since we're not verifying PoW any more, just blindly trust it, not verifying but blindly trusting the many year old merge point seems just the same trust model.
• Our tests still need to be able to generate pre and post merge blocks, so they need a new way to split the merge without TTD. The PR introduces a settable ttdBlock field on the consensus object which is used by tests as the block where originally the TTD happened. This is not needed for live nodes, we never want to generate old blocks.
• One merge transition consensus test was disabled. With a non-operational TD, testing how the client reacts to TTD is useless, it cannot react.

Questions:

• Should we also drop total terminal difficulty from the genesis json? It's a number we cannot react on any more, so maybe it would be cleaner to get rid of even more concepts.

[holiman]

Note, this is a backwards-incompatible change. Once blocks without TD have been written, users cannot switch back to an earlier version, right?
And if so, perhaps we need to version-bump after merging this PR ?

[holiman]

Also bump the db version number?

[rjl493456442]

If so, i would like to include my freezer pr as well. It also needs a db version bump

[karalabe]

It is 100% a one directional move, we should bump both.

[holiman]

Hope you don't mind if I reopen this? Pretty sure we want it...

[rjl493456442]

Yeah, we need it and i will review it next week.

[rjl493456442]

Yep, I can add it back Thanks and Best regards Gary rong Martin HS ***@***.***>于2024年11月18日 周一下午3:43写道：

…

***@***.**** commented on this pull request. ------------------------------ In core/rawdb/database.go <#30744 (comment)> : > @@ -409,7 +408,8 @@ func InspectDatabase(db ethdb.Database, keyPrefix, keyStart []byte) error { case bytes.HasPrefix(key, blockReceiptsPrefix) && len(key) == (len(blockReceiptsPrefix)+8+common.HashLength): receipts.Add(size) case bytes.HasPrefix(key, headerPrefix) && bytes.HasSuffix(key, headerTDSuffix): - tds.Add(size) + // The TD has been removed, but we still count it here IMO better leave the tds.Add too, so we actually count the data, not just ignore it. Otherwise the numbers won't match up — Reply to this email directly, view it on GitHub <#30744 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNO6OIOY2GCK5CFQE73OOT2BGLD5AVCNFSM6AAAAABRSNXYNWVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDINBRG42TIMRXGM> . You are receiving this because your review was requested.Message ID: ***@***.***>

[rjl493456442]

+-----------------------+---------------------------+------------+--------+
|       DATABASE        |         CATEGORY          |    SIZE    | ITEMS  |
+-----------------------+---------------------------+------------+--------+
| Key-Value store       | Headers                   | 576.00 B   |      1 |
| Key-Value store       | Bodies                    | 44.00 B    |      1 |
| Key-Value store       | Receipt lists             | 42.00 B    |      1 |
| Key-Value store       | Difficulties (deprecated) | 212.00 B   |      3 |
| Key-Value store       | Block number->hash        | 453.00 B   |      3 |
| Key-Value store       | Block hash->number        | 11.19 MiB  | 286075 |
| Key-Value store       | Transaction index         | 0.00 B     |      0 |
| Key-Value store       | Bloombit index            | 5.82 MiB   | 141382 |
| Key-Value store       | Contract codes            | 1.30 MiB   |    743 |
| Key-Value store       | Hash trie nodes           | 11.45 MiB  |  95278 |
| Key-Value store       | Path trie state lookups   | 0.00 B     |      0 |
| Key-Value store       | Path trie account nodes   | 0.00 B     |      0 |
| Key-Value store       | Path trie storage nodes   | 0.00 B     |      0 |
| Key-Value store       | Verkle trie nodes         | 0.00 B     |      0 |
| Key-Value store       | Verkle trie state lookups | 0.00 B     |      0 |
| Key-Value store       | Trie preimages            | 547.13 KiB |   8893 |
| Key-Value store       | Account snapshot          | 1.69 MiB   |  36542 |
| Key-Value store       | Storage snapshot          | 2.52 MiB   |  35341 |
| Key-Value store       | Beacon sync headers       | 0.00 B     |      0 |
| Key-Value store       | Clique snapshots          | 0.00 B     |      0 |
| Key-Value store       | Singleton metadata        | 697.91 KiB |     11 |
| Light client          | CHT trie nodes            | 14.92 MiB  | 139827 |
| Light client          | Bloom trie nodes          | 237.77 KiB |   1777 |
| Ancient store (Chain) | Headers                   | 82.31 MiB  | 286075 |
| Ancient store (Chain) | Hashes                    | 10.37 MiB  | 286075 |
| Ancient store (Chain) | Bodies                    | 33.41 MiB  | 286075 |
| Ancient store (Chain) | Receipts                  | 10.95 MiB  | 286075 |
+-----------------------+---------------------------+------------+--------+
|                                   TOTAL           | 187.37 MIB |        |
+-----------------------+---------------------------+------------+--------+

The db inspector has been fixed. Check the output format above

[fjl]

We need to ensure that downgrading to a previous version of Geth remains possible. The change will be backwards-incompatible if we delete the TD freezer table. So we should keep this table and write zero for now. At the next major release, v1.15.0, we can drop the TD freezer table.

The downgrade path needs to be tested w.r.t. this change to database, and peering with older versions also needs to be tested.

[barnabasbusa]

Should we also drop total terminal difficulty from the genesis json? It's a number we cannot react on any more, so maybe it would be cleaner to get rid of even more concepts.

lots of other clients also rely on genesis.json file to init a custom network, like erigon and reth. Even if you drop this value, we would need to coordinate with other clients that also use genesis.json format to also drop this value.

[fjl]

@barnabasbusa 'dropping' just means we would ignore it.

[rjl493456442]

Tests are failing, i will fix them next week

[rjl493456442]

@fjl I still prefer to bump the database version for this PR.

For freezer TD table, it's for sure we can write zero into it, however, once we downgrade
to legacy version, the TD in key-value store won't be found and blockchain will report
return consensus.ErrUnknownAncestor.

Personally I think it's not worthwhile to add the complexity for backward compatibility
and we can include it in the next major release.