Behavior change: produce a non-zero exit code (4) if no tests were run

[sunshowers]

Motivation

All but the simplest of projects have at least one test, and expect each invocation of nextest to run at least one test. Running zero tests likely indicates some kind of configuration issue.

Note that this is not a real error, so we're choosing to use the advisory exit code 4. Exit code 1 is for unexpected errors, 2 is for parsing errors, and 3 often has a specific meaning as well. 4 is the smallest unreserved code and fits this use case well.

Per our stability policy, this counts as a behavior change and will follow the rules outlined there, with a 3 month period. I don't anticipate any particular issues here -- some CI might break but it'll be correctly so.

Note that the behavior change is only for cargo nextest run, not list.

Update 2024-08-19

Based on feedback, we're going to make this a CLI option called --no-tests with three possible values:

• pass: same behavior as nextest 0.9.72 and below
• warn: new behavior, temporary default
• fail: new behavior, future default

The --no-tests=warn and pass is meant for cases like cargo hack --feature-powerset nextest run, where some combinations of features can lead to no tests being run in an expected manner.

See #1648.

Timeline

• 2024-08-18: Start warning about this with cargo-nextest 0.9.74
• After 2024-11-18: change behavior.

[sunshowers]

The behavior has been changed in #1881. I'll keep this issue until a bit longer after the next nextest release.

[sunshowers]

This behavior change has been implemented as of cargo-nextest 0.9.85. I did a brief survey of some GitHub issues and it looks like there was some minor breakage, but in general people seemed to understand the rationale behind the change.

[LeoniePhiline]

Hi!

The --no-tests flag is the only one requiring a = between flag and value (e.g. --no-tests=warn, while --no-tests warn is rejected).

All other nextest flags use a space.

It's not a big deal at all, but do you think this flag style could be unified?

Edit: Opened #1933 about that, to capture the issue more formally.