Investigate CI tests exceeding max execution time

[TomJGooding]

I've noticed recently there have been a number of CI failures where "The job running on runner GitHub Actions 35 has exceeded the maximum execution time of 360 minutes."

It looks like it always fails on the same test_notifications_loading_overlap_order test and only on Windows 10 or 11. There's a long timeout on the toast notification in the test app, but I wouldn't have thought that would impact the snapshot testing?

Example run: https://github.com/Textualize/textual/actions/runs/7071771658

Might be worth investigating...?

[github-actions]

Thank you for your issue. Give us a little time to review it.

PS. You might want to check the FAQ if you haven't done so already.

This is an automated reply, generated by FAQtory

[willmcgugan]

That is a strange one. It obviously does sometimes pass. The long timeout does jump out as a possible issue. Maybe the test code is waiting for the timer.

[willmcgugan]

I merged a change that dropped that notify timeout. Not sure yet if that fixed it, or it is still flakey.

Whoever picks this up: suggest you run the tests a few times on CI. To see if it reoccurs, before investigating further.

[TomJGooding]

It looks like that might have done the trick looking at the actions run today.

I'm still curious why it seems tests only sometimes might have been waiting on the timeout though!

[rodrigogiraoserrao]

Looking at all CI failures between the time of writing this and the time of Will's comment about the pushed fix (~75 CI failures in total), I didn't see another instance of this issue with timeouts and test_notifications_loading_overlap_order.

The closest thing I found were workflow runs #6322 and #6323 where all runs (in the OS / Python version matrix) timed out at random points of the testing which is a clear anomaly but also likely due to some malfunction of GH Actions or a couple of test runs that didn't get cancelled properly.

No other CI failure investigated was due to timeouts.

For reference, in this time period I investigated there have also been ~125 CI runs that completed all tests successfully, so I conclude that the flakiness of this test has likely been solved, especially taking into consideration how recurring this issue was in the test runs prior to Will's fix.

[github-actions]

Don't forget to star the repository!

Follow @textualizeio for Textual updates.