Revisit flaky tests

[willmcgugan]

We have some flaky tests. It seems to be a handful that are particularly sensitive to timing. I suspect that the issue lies in the tests themselves, and there is some genuine issue there. Let's look at those tests and see if we can make them stable.

[rodrigogiraoserrao]

Here's a survey of flaky tests on the 80-90 most recent CI failures, up to two months ago:

• tests/test_animation.py::test_schedule_reverse_animations, line 160 Example failing workflow

FAILED tests/test_animation.py::test_schedule_reverse_animations - assert (255, 255, 255) == (0, 0, 0)
  At index 0 diff: 255 != 0
  Full diff:
  - (0, 0, 0)
  + (255, 255, 255)

---

• tests/test_animation.py::test_scheduling_animation, line 55 Example failing workflow

FAILED tests/test_animation.py::test_scheduling_animation - assert (255, 255, 255) == (0, 0, 0)
  At index 0 diff: 255 != 0
  Full diff:
  - (0, 0, 0)
  + (255, 255, 255)

---

• FAILED tests/test_screen_modes.py::test_inactive_stack_is_alive - textual.css.query.NoMatches: No nodes match Example failing workflow

---

• tests/test_tabs.py::test_remove_tabs Example failing workflow

 +  where None = Tabs().active_tab

---

• tests/test_tabs.py::test_remove_tabs_messages, line 451 Example failing workflow

FAILED tests/test_tabs.py::test_remove_tabs_messages - AssertionError: assert ['on_tabs_tab...tabs_cleared'] == ['on_tabs_tab...tabs_cleared']
  At index 2 diff: 'on_tabs_cleared' != 'on_tabs_tab_activated'
  Right contains 2 more items, first extra item: 'on_tabs_tab_activated'
  Full diff:
    [
  -  'on_tabs_tab_activated',
  -  'on_tabs_tab_activated',
     'on_tabs_tab_activated',
     'on_tabs_tab_activated',
     'on_tabs_cleared',
    ]

---

• tests/snapshot_tests/test_snapshots.py::test_textual_dev_border_preview Example failing workflow

---

• tests/snapshot_tests/test_snapshots.py::test_textual_dev_colors_preview Example failing workflow

---

• tests/snapshot_tests/test_snapshots.py::test_command_palette Example failing workflow

---

• tests/tree/test_directory_tree.py::test_directory_tree_reload_other_node, line 158 Example failing workflow

FAILED tests/tree/test_directory_tree.py::test_directory_tree_reload_other_node - AssertionError: assert 0 == 1
 +  where 0 = len(<textual.widgets._tree.TreeNodes object at 0x000001D2D0C95A00>)
 +    where <textual.widgets._tree.TreeNodes object at 0x000001D2D0C95A00> = TreeNode('otherdir', DirEntry(path=WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_directory_tree_reload_oth0/otherdir'), loaded=True)).children

---

• tests/tree/test_directory_tree.py::test_directory_tree_reload_node, line 120 Example failing workflow

FAILED tests/tree/test_directory_tree.py::test_directory_tree_reload_node - AssertionError: assert 1 == 2
 +  where 1 = len(<textual.widgets._tree.TreeNodes object at 0x000001D8C7CA16D0>)
 +    where <textual.widgets._tree.TreeNodes object at 0x000001D8C7CA16D0> = TreeNode('parentdir', DirEntry(path=WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_directory_tree_reload_nod0/parentdir'), loaded=True)).children

---

• tests/notifications/test_app_notifications.py::test_app_with_notifications_that_expire, line 40 Example failing workflow

FAILED tests/notifications/test_app_notifications.py::test_app_with_notifications_that_expire - AssertionError: assert 50 == 100
 +  where 50 = len(<textual.notifications.Notifications object at 0x112c66cd0>)
 +    where <textual.notifications.Notifications object at 0x112c66cd0> = NotificationApp(title='NotificationApp', classes={'-dark-mode'})._notifications
 +      where NotificationApp(title='NotificationApp', classes={'-dark-mode'}) = <Pilot app=NotificationApp(title='NotificationApp', classes={'-dark-mode'})>.app

[darrenburns]

@rodrigogiraoserrao Thanks for doing this summary - it's really helpful!

[github-actions]

Don't forget to star the repository!

Follow @textualizeio for Textual updates.