Test is not reported if conditionally disabled

[kaselis]

What happened?

Conditionally disabling test (other than "Disabled" annotation) makes it completely disappear from reports.

What did you expect to happen?

Report would show it either ignored or skipped.

Serenity BDD version

4.2.8

JDK version

21

Execution environment

Mac OS

How to reproduce the bug.

Cloned junit starter and disabled one test conditionally - https://github.com/kaselis/serenity-junit-starter. Just running ./mvnw clean verify produces a report with 4 passing tests and 0 other kinds.

How can we make it happen?

Add it to the Serenity BDD backlog and wait for a volunteer to pick it up

[wakaleo]

That's not currently supported - if JUnit doesn't run the test, Serenity has no way of picking it up.

[kaselis]

Junit does run it and it reports as disabled. What I was able to figure out, SerenityTestExecutionListener class receives that the test was skipped, and then it proceeds to check if it has "Disabled" annotation. If it does - serenity reports correctly as ignored - https://github.com/serenity-bdd/serenity-core/blob/main/serenity-junit5/src/main/java/net/serenitybdd/junit5/SerenityTestExecutionListener.java#L195 . If it doesn't find that annotation, the test "result/execution" (I understand that test is not executed) vanishes. I am wondering, instead of ignoring those tests, would it make sense to add them to skipped? Junit invokes this method in the listener class - public void executionSkipped(...)

[wakaleo]

Yes it would. That feature of Junit is new to me, so isn't factored into the current code, but there's no reason why it shouldn't be.

[kaselis]

I am wondering, if isIgnored() check removed from here, would that break anything? And I think it would be enough to account for those disabled tests. In my understanding, if the execution got to that point, it is because junit reported the test as being skipped/ignored anyways.