Promote zopen_test into a new high-level verb

[v1gnesh]

... called zopen-test (although /run tests is available for manual invocation).
This will help us get granular reporting for CI and also doing just one if only one is necessary.

Further, what about wrapping the various tests up in bats?

Thoughts? 🤔

[DevonianTeuchter]

Did start playing with shunit2 as it supports sh, bash, zsh (and ksh should anyone use that!) so supports all the shells on zos. Seemed a bit more usable than bats tbh but not really had the time to investigate properly!

[v1gnesh]

Main thing I'm thinking of is just splitting build_and_test.yml into build.yml and test.yml, allowing slightly more granular pass/fail badges in PRs (where we see the build pass tick), and setting the stage for similar expansion (ex: unit test, sec test, whatever).

[v1gnesh]

Would it run the check step in zopen build?

To start off, zopen test would just do what zopen_test() function in buildenv does today.
And zopen build wouldn't/shouldn't do it anymore.
EDIT: Ah yes, totally forgot about check. Yup, just pulling that out.

Or would it run the tests which are part of metaport?

Hmm... so a theoretical zopen test should run the port's tests as well as test meta... ?

[IgorTodorovskiIBM]

Typically the tests are integrated into the build. So you need to go through the step of bootstrap, configure, build, and then test. zopen test could do all of those and disable install at the end.

[v1gnesh]

Hmm.. I've only seen a handful of build instructions in the wild.
They're all like
Do make x to build
Do make test to run tests
Do make release to create releases
Do make release-specific1 for create a specific item
etc.

So in that way, make test would expect the binary to be available somewhere (or perhaps it builds first, by having a pre-req on build as you say), and is just to test it out.
Keeping it separate also means we have the choice of testing only major releases, or some such.

[IgorTodorovskiIBM]

Most of the autotools based projects use make check. For what you're proposing, make installcheck may be better. I opened an issue on this awhile back: #598.
However, I'm not sure at this point how many tools specify an installcheck target - https://www.gnu.org/software/automake/manual/html_node/Basic-Installation.html