new ReducingMethod KEEP_ONLY_LATEST_SCENARIO_RUNS #914
Conversation
add a new reducer that can keep only the latest scenario run. useful for times when you have flaky scenarios and you do retries on the given scenario, but don't want the flaked attempts to appear in report.
| @@ -82,6 +90,42 @@ public ReportParser(Configuration configuration) { | |||
|
|
|||
| return featureResults; | |||
| } | |||
|
|
|||
| /** | |||
| * If the JSON file has the same scenarios run multiple times, keep only the | |||
There was a problem hiding this comment.
AFAIR JSON does not have results for multiple times. For that case you would rather have more than one JSON file. Can you explain this case?
There was a problem hiding this comment.
There was a problem hiding this comment.
@damianszczepanik - I know this is an old PR. This case happens when we use TestNGCucumberRunner and use TestNG's retry analyzer capability to retry tests to disregard flaky tests.
Assuming that we run 1 test with re-run enabled -
What this does is, in case of flaky behavior, where a test would fail first and then would pass when re-tried - it creates 2 entries in JSON for the same test - 1 for the failed test and 2nd for the same test, re-tried. So the final report shows that there were 2 tests.
If the re-tried scenario also fails, it is counted as another failed test. so total 2 failed tests
If the re-tried scenario passed, the report says, 1 failed and 1 passed.
This PR seems to solve that problem.
There was a problem hiding this comment.
Exactly. Sorry I didn't respond earlier, got moved on to other priorities and never had a chance to come back and answer the questions/fixes.
| } | ||
| } | ||
|
|
||
| private void addRetryNumberToElmIfNeeded(Optional<Element> lastElement, int numRemoved) { |
There was a problem hiding this comment.
this is Java, we don't shorten names to ElmI
| @@ -15,7 +15,7 @@ | |||
|
|
|||
| // Start: attributes from JSON file report | |||
| private final String id = null; | |||
| private final String name = null; | |||
| private String name = null; | |||
There was a problem hiding this comment.
i change this to mutable because i use the Name in the report because I append " (${numRetries)" so that it shows up in the report.
alternatively we could create a new element in the html/xslt or whatever.
|
when will this feature be available in plugin? |
|
This is a super useful update. +1 to get it in as soon as possible. |
|
it has conflicts with master branch |
|
OK I will get the conflicts fixed and the issues resolved so we can get this merged. ASAP. |
|
Hi Team, can anyone please resolve the conflicts for the ^ fix, I am facing a similar issue whereing
I would really appreciate if someone can point me in the right direction if the issue has already been fixed. |
|
@nddipiazza @damianszczepanik Greetings. Can someone please resolve the merge conflicts and get this PR merged please ? |
|
Hi Team, |
|
I am seeing the behaviour described by @pawarvaibhav |
add a new reducer that can keep only the latest scenario run.
useful for times when you have flaky scenarios and you do retries on
the given scenario, but don't want the flaked attempts to appear in
report.