Subsuite status FAILED based on verdict/outcome #61
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andmat900
requested review from
t-persson and
fredjn
and removed request for
a team
andmat900
force-pushed
the
20240527_custom_verdicts
branch
from
3d42803 to
03e4079
Compare
t-persson
reviewed
May 30, 2024
| return self._failed | ||
| verdict = self.outcome().get("verdict") | ||
| conclusion = self.outcome().get("conclusion") | ||
| return "FAILED" in (verdict, conclusion) |
There was a problem hiding this comment.
I don't think we should set failed if the verdict is failed. Verdict does not mean that the sub suite failed. It could be a genuine verdict. The verdict of sub suites should be evaluated in the results method in the Suite class. The result will be exactly the same, but it will allow us to do more in the future without editing up here.
There was a problem hiding this comment.
This change won't be needed if we proceed with the current proposal of exit code handling/verdict matching in the Test Runner:
https://github.com/eiffel-community/etos-test-runner/pull/49/files
fredjn
approved these changes
Jun 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Applicable Issues
Description of the Change
This change modifies the behavior of the "failed" instance variable. If not set explicitly, it will return true if either verdict or conclusion of the suite is set to FAILED.
Without this change Suite Runner does not handle a case if a sub suite verdict and conclusion are both set to FAILED. In that case the verdict description is: "No description received from ESR or ETR." With this change it is set to "{len(failed)} sub suites failed to start"
Alternate Designs
Possible Drawbacks
Sign-off
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
Signed-off-by: Andrei Matveyeu, [email protected]