Save screenshot on App.run_test failure

[ab-10]

Suggestion to add an option screenshot_on_fail: Optional[Path|str] = None in App.run_test to save a screenshot of the app upon a test failure and error.
Happy to implement the feature myself, if it aligns with the project's goals.

Example functionality

async with app.run_test(screenshot_on_fail="logs/") as pilot:  
    # Test pressing the R key
    await pilot.press("r")  

    assert app.screen.styles.background == Color.parse("red")

If the assertion would fail, this would save a screenshot of the app at the time of the failure in logs/textualize-<timestamp>.svg

Open questions

1. How to enable the user to configure the save location and filename for the tests? What would be reasonable defaults here?
2. Do other contributors find such functionality desirable?

[willmcgugan]

Interesting idea. Can you describe the kind of tests you might want this feature for?

@davep @rodrigogiraoserrao thoughts?

[ab-10]

Can you describe the kind of tests you might want this feature for?

I think all UI tests could benefit from this feature. In particular, I hope for this feature to provide a straightforward way to capture the application state, enabling the developer to understand what assumptions about the UI were incorrect without a lengthy debugging process. For example in the tutorial code test_rgb.py <https://textual.textualize.io/guide/testing/#testing-apps>, `test_buttons` might fail because of two main reasons: 1. The runner is unable to find the specified element (e.g. `await pilot. click("#red")` fails), causing an error. 2. The series of actions produce an incorrect state (e.g. `assert app.screen .styles.background == Color.parse("red")` fails), causing a test failure. Having a screenshot would enable the developer to determine that: 1. The logic for loading the button didn't work or it did work but the CSS selector in the test was incorrect. 2. What exactly was the color of the background?

…

On Mon, Dec 4, 2023 at 2:30 PM Will McGugan ***@***.***> wrote: Interesting idea. Can you describe the kind of tests you might want this feature for? @davep <https://github.com/davep> @rodrigogiraoserrao <https://github.com/rodrigogiraoserrao> thoughts? — Reply to this email directly, view it on GitHub <#3702 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC54L5RURNA4TJ2R4LFR7FTYHXNADAVCNFSM6AAAAAA7RDAGWSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZYG42TINJRGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[rodrigogiraoserrao]

@davep @rodrigogiraoserrao thoughts?

This looks helpful to debug tests.
It's something that could even be turned off by default and we just turn on locally when a test is failing for some reason we don't understand.

[davep]

I'm not opposed to the idea, although I'm struggling to see the utility in it; the couple of examples given above seem to be like things I wouldn't ascertain from a screenshot anyway (you can't see IDs in a screenshot, for example).

[rodrigogiraoserrao]

I'm not opposed to the idea, although I'm struggling to see the utility in it; the couple of examples given above seem to be like things I wouldn't ascertain from a screenshot anyway (you can't see IDs in a screenshot, for example).

Maybe the example isn't great but I've found myself wishing for this feature in the past, fyi...
Maybe that's not enough to implement it, but I would've used it in the past.

[ab-10]

Thank you @willmcgugan, @rodrigogiraoserrao, and @davep for commnents. The feature is a bit more controversial than I initially expected. I will close this issue, especially since I expect the actual implementation to have a couple of subjective elements in it as well (e.g. how to configure the storage path for the images).

[github-actions]

Don't forget to star the repository!

Follow @textualizeio for Textual updates.

[github-actions]

Don't forget to star the repository!

Follow @textualizeio for Textual updates.