Github Status reported Failure even though tests passed

[nsoojin]

Troubleshooting

• I've searched discuss.bitrise.io for possible solutions.
• Which version of the step is effected? 4.3
• Is the issue reproducible with the latest version? YES
• Does the issue happen sporadically, or every time? EVERY TIME
• Is the issue reproducible locally by following our local debug guide? NOT APPLICABLE

Useful information

• FAQ
• devcenter.bitrise.io

Issue description

I have some flaky tests that sometimes fail. So I'm using "Retry on Failure" option for my test plan. However, Bitrise is reporting tests failed even though flaky tests passed on subsequent tries.

XCResult here shows that the overall test suite passed with 614 passed tests, 0 failed tests, and 2 mixed tests which are the flaky tests that passed on the second try. However, Bitrise's Test result marks them "failed", so the status for my PR failed.

Expected Result

If retried tests passed, the overall test should be marked successful and Github Status should pass.

Bitrise info

• Build URL: https://app.bitrise.io/build/1d48a2ba-5a42-486f-b81b-077639086396?tab=details
• Bitrise Support enabled: _YES
• Log:

[DamienBitrise]

The test report addon does seem to be reporting failures even tho the step passed. Thanks for the report, we will look into this issue.

[d4rkd3v1l]

We're facing the same issue, I guess. (step version 4.6)

When a test fails initially, but then passes on retry, the build is passed in Bitrise (as expected), but Bitrise Checks reports a failed status to GitHub.

[nsoojin]

@DamienBitrise Can you share us an update?

[bitrise-coresteps-bot]

Hello there, I'm a bot. On behalf of the community I thank you for opening this issue.

To help our human contributors focus on the most relevant reports, I check up on old issues to see if they're still relevant.
This issue has had no activity for 90 days, so I marked it as stale.

The community would appreciate if you could check if the issue still persists. If it isn't, please close it.
If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me".

If no comment left within 21 days, this issue will be closed.

[nsoojin]

still important to me

[bitrise-coresteps-bot]

Hello there, I'm a bot. On behalf of the community I thank you for opening this issue.

To help our human contributors focus on the most relevant reports, I check up on old issues to see if they're still relevant.
This issue has had no activity for 90 days, so I marked it as stale.

The community would appreciate if you could check if the issue still persists. If it isn't, please close it.
If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me".

If no comment left within 21 days, this issue will be closed.

[lo1tuma]

Looks like this is duplicate of #195. It seems like only the test report and the github status check is wrongly reported as failed, the bitrise step itself is passing. This is a huge problem for us, because we rely on the github status check which must pass before we allow pull requests being merged.

Is there any update on this bug?

[bitrise-coresteps-bot]

Hello there, I'm a bot. On behalf of the community I thank you for opening this issue.

To help our human contributors focus on the most relevant reports, I check up on old issues to see if they're still relevant.
This issue has had no activity for 90 days, so I marked it as stale.

The community would appreciate if you could check if the issue still persists. If it isn't, please close it.
If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me".

If no comment left within 21 days, this issue will be closed.

[lo1tuma]

This is still an issue.

[bitrise-coresteps-bot]

Hello there, I'm a bot. On behalf of the community I thank you for opening this issue.

To help our human contributors focus on the most relevant reports, I check up on old issues to see if they're still relevant.
This issue has had no activity for 90 days, so I marked it as stale.

The community would appreciate if you could check if the issue still persists. If it isn't, please close it.
If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me".

If no comment left within 21 days, this issue will be closed.

[bitrise-coresteps-bot]

I'll close this issue as it doesn't seem to be relevant anymore.
We believe an old issue probably has a bunch of context that's no longer relevant, therefore, if the problem still persists, please open a new issue.