The big ol' compedium of flaky tests

[SquidDev]

CC:T's test suite is incredibly flaky when run via CI. I think this is just a limitation of how powerful the runners are (they work fine on my machine!) - but having to rerun every third build gets old quickly.

This issue is an attempt to track recent tests failures.

Gametests fail due to timeout

Builds

• Build 2243: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4413979109/jobs/7735111084
• Build 2257: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4430925396/jobs/7773273120, https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4430925396/jobs/7773650001 (only some failed here)
• Build 2271: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4525631854/jobs/7970262900

Notes

In 2243, the test batch was created at 09:21:22 and finally started running at 09:21:25 (this is 3s for a single tick, which is Bad(TM)). Computers continue to turn on until 09:21:28, and then tests time out at 09:21:29 (after 150 ticks). There's two odd things here:

• Computers really shouldn't take 3 seconds to all turn on - we're using the fake Kotlin machine here, which has very little startup overhead.
• Tests time out after 150 ticks, which should take 7.5s, but only appears to be 4s. Unless Minecraft is skipping ticks here or something - maybe we should add some logging here just for debugging?

modem_test.have_peripherals.modem_test.have_peripherals

Starts with peripherals ==> 
 expected: <[monitor_0, printer_0, right]> 
  but was: <[right]> at 4,-58,22 (relative: 4,2,2) (t=1)

Builds

• Build 2257: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4430925396/jobs/7773927203
• Build 2268: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4515339240/jobs/7952515237

inventory_test.fails_on_full.inventory_test.fails_on_full

No peripheral: minecraft:furnace_0 at 47,-58,11 (relative: 2,2,1) (t=1)

Builds

• Build 2268: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4515339240/jobs/7952515237

modem_test.full_modems_form_networks.modem_test.full_modems_form_networks

On the same network ==> expected: <dan200.computercraft.impl.network.wired.WiredNetworkImpl@554b3d66> but was: <dan200.computercraft.impl.network.wired.WiredNetworkImpl@c4809f9>

Builds

• Build 2268: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4515339240/jobs/7952515237

turtle_test.peripheral_change.turtle_test.peripheral_change failed

expected: <[(peripheral_detach, right), (peripheral, right)]> 
 but was: <[(peripheral_detach, right)]>

Builds

• Build 2270: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4525430871/jobs/7969917473

[a2937]

Pardon me, but where does the tests live and how do you run it?

[SquidDev]

The game tests live in the projects/common/src/testMod folder and can be run with ./gradlew runGametest.

The contributing and architecture docs have a little more info.