Run fast/slow tests first

[fenollp]

Hello!
I'd like the queue of tests to run to be ordered by the time each took last time nextest was run. It's a simple and weak heuristic with little impact on the call.

Maybe this can be opt-in with a flag a la --ordering=fast-first

[sunshowers]

Hi there --

This would be awesome to have, the only issue is that we don't record previous test runs at the moment. There's some design work that needs to happen to support that.

[sunshowers]

#958 is about running the slowest tests first, which is something that would also be useful.

[HadrienG2]

In the process of implementing this, would it be possible to publicly expose the ability to dump the execution statistics that nextest displays during execution in a machine-friendly format ? I'd like to have that data sorted by decreasing time as a kind of execution profile to know which tests I should optimize first.

[sunshowers]

@HadrienG2 I think that would be part of #20 if I'm understanding you correctly.

[HadrienG2]

Probably, I was just thinking that implementing this feature requires persistifying exactly the data I want ;)

[torokati44]

Since #1152 has started progressing a little, can we get some guidence as for what's the next blocker for this? 😶
As hinted to by the referring PR above, we'd be interested in running the tests in decreasing order of duration. It significantly improves multi-core utilization, therefore decreases overall testing time.

[matthiasgoergens]

As a stop-gap, would it be possible to run tests in random order?

[torokati44]

As a stop-gap, would it be possible to run tests in random order?

We tried that:
https://github.com/ruffle-rs/ruffle/actions/runs/10902467131/job/30254429769?pr=17961#step:9:444

But not really:

nextest/nextest-runner/src/list/test_list.rs

Line 635 in a4490cd

list.sort_unstable();