Feature request: allow multiple -t arguments / add a --all argument

[me-and]

Currently it's only possible to run a single test at a time, at least without multiple invocations of git-test. It'd be useful to me to be able to specify multiple tests to run at the same time. I'd see this working like so:

• git test run <commit>...: Runs the default test on the specified commits.
• git test run -t alpha -t beta <commit>...: Runs the tests alpha and beta on the specified commits.
• git test run -t default -t alpha <commit>...: Runs the tests default and alpha on the specified commits.
• git test run --all <commit>...: Runs all the defined tests, including default if defined, on the specified commits.

There's a small-but-greater-than-zero chance I'll manage a PR myself for this, but given the other projects currently on my plate I don't want to promise anything.

[veo-jdowner]

I wanted something similar and hacked in a solution the other day (https://github.com/veo-jdowner/git-test/tree/working). It is obviously not the right solution but it would be good to get some guidance @mhagger on how you would like it added if you support this feature.

In my case, I wanted syntax like,

git test run --test=foo,bar,baz ...

but the functionality is the important thing.

[mhagger]

Thanks for your comments and the spike implementation! Sorry I haven't been more responsive but I've been very busy lately.

I think this is a great idea and it's one that I'd use myself. Let's talk a little bit about the requirements. When multiple tests are specified:

• git test run should run each of the tests, in the order specified, against each of the commits specified. A commit must pass every one of the tests for it to be considered to pass as a whole. As soon as a commit fails one test, that commit should be considered to have failed, and no more tests should be run against it (unless --keep-going was specified). The results of each test should be recorded individually, test by test. If any of the tests has already been run against one of the commits' trees, then its old result should usually be reused. If one of the specified tests has already failed against the current commit, then normally that result should be printed and no tests should be run against that commit.
  If --retest is specified, then that option should only apply to any tests that have stored failing results. For example, if I'm running tests a, b, and c, and the current tree has a success result stored for test a, a failing result stored for test b, and no result stored for test c, then tests b and c should be run but not a. On the other hand, --force and --forget should apply to all tests.
  If --keep-going is specified and a test fails against a certain commit, then the other tests should nevertheless be run against that commit before proceeding to subsequent commits.
  git test run and git test results should display the results for each test separately. Especially, if there is a failure, git test run should make it obvious which test failed. It might be nice for git test results to list the results in a table with one line per commit rather than printing one line per (tree, test) pair.

• git test add should error out if more than one test is specified.

• git test forget-results should forget the results for all tests that are specified. (We could live without this feature.)

• git test remove should remove all of the tests that are specified. (We could live without this feature.)

I don't have a preference between -t test1 -t test2 vs. -t test1,test2. In fact, it wouldn't be hard to support both syntaxes and even -t test1 -t test2,test3 by having the option append arg.split(',') to a list.

As for how to implement it, my first inclination is to make Test into an abstract base class, with one implementation being a NamedTest that has the current behavior. But there would also be a CompoundTest implementation, which contains an array of Test describing the subtests. There would be a new function like make_test() that returns a single NamedTest if only one case was specified, or a CompoundTest if more than one. The CompoundTest class would implement many of its methods by looping over its array of subtests and calling the corresponding method on each one, combining the results as appropriate and returning them to the original caller. In this way it might be possible to add support for multiple tests without having to change much of the existing code.

I haven't checked carefully whether this will really work. Can all of the Test methods really be implemented by CompoundTest and give the desired behavior? It might be necessary to add a couple more methods to the Test interface to get the right semantics.

It would be great if somebody wants to work on this. Realistically, I don't expect to have time for it anytime soon.