Feature req.: write the list of tests being ran to a file

[dpc]

This might be a bit of XY problem so first the motivation:

Multiple times in last year in our CI we had (well... usually I did) introduced bugs that lead us to believe we are running all the tests while we were running none/subset of them. This wrecks havoc in the development process. We rely on the CI to enforce that the tests are passing and since it's Rust oftentime developers have no reasons to even run the tests locally, as "if it compiles, it usually works". That's all fine - if the tests are actually being ran in the CI.

It's a pain to recover from such events, because it's much easier to figure out any regression while working on a change, than days/weeks afterwards after we've figured out what happened and need to scramble fixing all the bugs the were not detect by the CI.

Latest breakage was caused by #1108 , but it's not a cargo-nextest specific problem. It happened for various reasons in the past.

This has become embarrassing so I asked around what do people do to prevent it once and for all, and the best idea I've got was: check-in the list of executed tests into git, and verify it against tests that were actually ran. I really like this idea, but I'm not sure if I can actually achieve it with existing functionality.

The way I imagine it is having some ./tests.list like file(s), checked in. In the CI rm ./tests.list, then all the tests script would run cargo-nextest with some --append-test-log ./tests.list, and as the last step of the CI, (possibly sort and massage that ./tests.list) and verify that the resulting ./tests.list is the same as the checked-in reference.

I'm open to other ideas and discussion.

[sunshowers]

Thanks for the suggestion! It's definitely thought-provoking -- it's kind of like a meta-level test that you're running the right set of tests.

Are there prior implementations of this anywhere?

[dpc]

I was pointed to https://github.com/rust-bitcoin/rust-bitcoin/blob/675da341274f7338fcf5a7c6ca9bd6b2f254b511/.github/workflows/fuzz.yml#L66