Milestone: Refactor Go DHT codebases

[iand]

ETA: 2023-09-30

Description: Refactor the current Go DHT implementation to make it easier to maintain, more reliiable and provides a solid foundation for future extension and the Composable DHT.

Deliverable: a working but unoptimised DHT that can be used in Kubo without error. Performance and resource consumption should be no more than 10% (?) worse than current DHT implementation, with the expectation that better results will come with optimization.

Motivation:

1. The codebase is very hard to understand and to modify. While trying to implement IPIP-373: Double Hash DHT Spec ipfs/specs#373, we figured that it would be more time-efficient to refactor/rewrite the implementation rather than trying to plug the changes in the existing implementation.
2. Flaky tests due to concurrency issues. Unit tests evaluating if the implementation is working as expected are impossible to implement without a major refactor due to massive parallelization. Having a sequential Kademlia implementation would enable proper testing.
3. Performance evaluation. For the reason above, it is hard to conduct performance evaluation tests. The current implementation is far from optimized from a resources POV, and it is likely the cause of Optimizing (or remove) the provider delay interval for DHT lookups used by Bitswap ipfs/kubo#8807, feat: set Bitswap.ProviderSearchDelay to 0s ipfs/kubo#9530, [Thunderdome Experiment Summary: provdelayrouting](https://www.notion.so/Thunderdome-Experiment-Summary-provdelayrouting-41b5807982c74938a9e6e15a3e38a7ad?pvs=21)). We cannot efficiently debug the current DHT implementation to find where the problem comes from. The outcome of feat: set Bitswap.ProviderSearchDelay to 0s ipfs/kubo#9530 is expected to have a high impact, as it could lead to a massive reduction in the Bitswap broadcast, but is depending on solving bugs in the DHT.
4. Hard (or even impossible?) to test the protocol. IIUC Testground was built to help with testing the DHT, but hasn’t been (very) successful so far. Having a sequential DHT, using a fake clock would make protocol testing, evaluation and simulations much easier. So research and enhancements of the Kademlia protocol would become easier.
5. Implementation mistakes. The current implementation contains multiple serious flaws that have been introduced over time. For instance, Kademlia should only handle Kademlia identifiers (256-bits bitstrings) internally, but it is currently using strings. A new start helps correcting all these fundamental issues, and allows us to apply the learnings gained over time.

More information in Notion

Children:

• Refactor go-libp2p-kad-dht to use go-kademlia libp2p/go-libp2p-kad-dht#855
• Minimally Functional Modular DHT zikade#17
• Create implementation of DHT using go-kademlia ipfs/boxo#408
• Integration Testing with Kubo zikade#18

[BigLep]

I agree with the motivation items that are listed, but I am thinking we could give some more user-focused set of statements first like:

1. Unlock the composable DHT effort (and link to representative to tracking issue), which is a win for libp2p users who need and want to "share" a DHT that supports "features" beyond IPFS and win for all users (including IPFS) since upgrades don't need to be synchronized.
2. Enable the double hashing DHT for reader privacy so that IPFS nodes have a reader privacy story for IPNI and DHT content routing stories.
3. Address the performance issues which have held back addressing the long-standing provider delay interval for content routing (DHT, IPNI) lookups used by Bitswap, which should reduce the time to first byte by hundreds of milliseconds when content is not available from a currently-connected bitswap peer.
4. Reduce ongoing maintenance costs because of flaky tests or elaborate testing setups with Testground.
5. Empower researchers to more easily self-service run experiments in a simulated environment or even as a new "feature" in the upgradable/composable DHT.

@iand - I would modify the description myself, but I don't have permission. Can I please be added to the repo? (This also isn't under github-mgmt which is why I didn't open a PR for this...)

[iand]

@BigLep added you to this repo. I will update the issue description, adding in references as you suggested