Improve bitswap performance

[dirkmc]

In particular we'd like to focus on improving bitswap performance for

• one node retrieving a large amount of data from several nodes who already have it
• one node retrieving a segment of data from several nodes who have different parts of it
• a group of nodes retrieving data from several far-away nodes who already have it

[dirkmc]

I created an issue in go-bitswap to explain the current implementation of bitswap sessions, as a first step towards understanding where improvements can be made.

[dirkmc]

I created an issue in go-bitswap to highlight some issues with latency measurement, and proposals to address them

[dirkmc]

I created an issue in go-bitswap to highlight some issues with session request sharding, and explore parameters for a solution.

[dirkmc]

I created an issue in go-bitswap to discuss randomizing want list requests.

[dirkmc]

I created a PR in go-bitswap to improve handling of response messages: it processes messages as a batch instead of one-by-one, which improves session request splitting.

[dirkmc]

This week

• merged PR to improve handling of response messages
• merged refactoring PR which will simplify bigger changes we'd like to make once we have worked out how to tackle request sharding (Expect to have this worked out early next week)
• opened PR to recognize when we've received a block that we want but haven't had a chance to ask for yet: Fix: don't ignore received blocks for pending wants ipfs/go-bitswap#174

[dirkmc]

Last week

• merged PR to recognize when we've received a block that we want but haven't had a chance to ask for yet: Fix: don't ignore received blocks for pending wants ipfs/go-bitswap#174
• opened issue to discuss RFC for Bitswap extensions: [Draft] Bitswap Protocol Extensions ipfs/go-bitswap#186

[dirkmc]

Last week

• Implemented PoC of Bitswap Protocol extensions
  This week
• Improving PoC performance against Bitswap benchmarks

[dirkmc]

FYI I was working on improving the Bitswap benchmarks by making small modifications to the PoC, but it was too haphazard and buggy, so I'm going to do a proper refactor first.

[dirkmc]

With the latest changes the PoC seems stable, so I'm looking into improving performance.

I created an issue to discuss how we can improve the performance of bitswap's engine (ie the part of bitswap that receives wants and sends out blocks).
I've implemented Proposals 1 - 4, which address ordering wants and grouping them by block size.
Next I will implement proposal 5, which adds parallelism.