Package Manager usecase benchmarks for #ipfs-test-infra project

[andrew]

Related to #76, #77 and #79

One of the Infrastructure/IPFSaaS team's goals this quarter is to be able to test potential IPFS releases against real-world usecase benchmarks to catch any potential regressions early.

Package managers are one of those usecases, we should package up some of those benchmarks for the @ipfs/wg-infrastructure team to integrate into their test suite.

[meiqimichelle]

We can also explore whether we should do this work as part of https://github.com/ipfs/benchmarks //cc @alanshaw

[alanshaw]

The test scenarios we have are documented here https://github.com/ipfs/benchmarks/tree/master/tests#nodejs-and-go - happy to talk with either of you about augmenting them for your needs

[andrew]

There's two kinds of things to think about here:

• tests - integration level tests that cover our usecases that either pass/fail
• benchmarks - measure the performance of running our usecases over time and ensuring there aren't regressions

[hsanjuan]

Note that @ipfs/wg-infrastructure is not working on test suites. I'm not sure the pipeline that is being setup by @jimpick or @alanshaw etc can accomodate adding TBs of data. Maybe you will need a separate pipeline. Also, worth checking if the problems seen with multiple repositories of 1TB can already be seen with test-sets of 1 GB.

The bottleneck is probably disk througput on write. This will become slower with flatfs and probably statys reasonably constant with badger (assuming enough memory). All the tests listed by @alanshaw probably grasp a lot of metrics which, if improved, will help adding large repos.

Having tracing for the "add" process is probably the first place to get an idea of what is taking longest.

[jimpick]

The existing ipfs/benchmarks setup is using a single bare hardware minion for the tests, I believe. But I'm also doing work with https://github.com/libp2p/testlab which sets up a cluster ... we could design a test target cluster with the resources required for some heavy tests.