Latent Actions cause tests to early terminate

[shwarnock]

I am having an issue when using the base Delay and SetTimerByEvent BP nodes in tests.
The issue presents itself only when I run the tests from the SessionFrontend. When I run any test with one of the above BP nodes, the first time the test is run after launching the editor, everything works fine. However, all subsequent runs of the test early terminates the test. It still gets marked as 'Passed' but if I place a breakpoint after the Delay or on the event attached to the SetTimerByEvent, the breakpoint is never hit. I also placed a breakpoint in C++ at FinishAct, and I never hit that breakpoint either.

I am running on 4.26.

[michaelWuensch]

Hi, did you find a solution? I have exactly the same problem running 4.25.
I want to achieve 2 things and both do not work, as the tests are terminated (with success state) after a delay greater than around 0.5 seconds.

The first thing I am trying to do is some interaction with a backend in the ACT phase, but if this takes to long, the test just passes altough finishACT is not yet called.

The second thing I want to achieve is to get some latent action work on "BeforeAllTests" and FINISH those action before the first test starts. I modified the code to have a "FinishBeforeAllTests" function, but then ran again into the skipping if it is more than around 0.5 seconds.

To replicate this, all you have to do is adding a 1 second delay in the ACT Event and then call finishACT. The test will then be passed successful even if it should fail. Unfortunatelly it does not wait for the timeout which is set to 30 seconds.

Any help would be appreciated.

[HunterOvadia]

Is there any update on this?

[Insythor]

I am currently experiencing the same issue (4.25). One work around I did find was if the test suites have tick turned OFF, they will not start to execute the tests (only works in PIE), so then you can create an AfterAll call to the test suits (added a tag "OFF" to search for them) to enable tick which starts the tests.

We have a project where we need to check about ~100 params, wait a few seconds, then check that they are turned off - this also has the issue of causing unreal to trigger the infinite loop error, as somehow the ~100 tests are calling assume events over 250 times so fast that it crashes PIE. It seems like the BeforeEach in the test suit is checking the assume event, when it should be called when the test is called (In my case it only needs to be called a single time in each test, not sure if there's a way around that)?

Also we have a lot of things which need to be on for a few seconds, then turn off, and we need to check that it remains on for the duration, which seemingly cannot be done as the test just finishes instantly.

If anyone using this has come up with a modification / work around for this "Feature" please post it! :D

Thanks.

[teddyk-pubg]

This issue is still valid, and the included test AlwaysSucceedWithDelay throws a false positive. Finish Act is never called and the Assertion either, so you cannot even cause the test to fail as it should in this circumstance (by asserting against the Delay Elapsed boolean for example).

Anyone make any progress?

[teddyk-pubg]

LatentCommand FTestAutomationPluginWaitForEndOfTestSuite::Update() needs to be changed to check that the Suite is actually finished, currently the exit condition is IsRunning(), so it will teardown as soon as the test suite starts. It can take a bit for the world to come up even after BeginPlay(), so if your test is fast it will run successfully- anything which takes longer than the world to load will fail.

In the process of accommodating this change, I had to refactor the TestSuiteActor entirely, there are many issues and a dirty recursive algorithm in there waiting to explode.