Snapshot testing guide (#3357)

* Snapshot testing guide * Typo fixes * Some more typo fixes * Typo fixes * Update docs/guide/testing.md Co-authored-by: Rodrigo Girão Serrão <[email protected]> * Add clarifications, PR feedback * Add clarifications, PR feedback --------- Co-authored-by: Rodrigo Girão Serrão <[email protected]>
Textualize · Sep 21, 2023 · 9c8a8df · 9c8a8df
1 parent bbde62f
commit 9c8a8df
Show file tree

Hide file tree

Showing 5 changed files with 144 additions and 2 deletions.
diff --git a/docs/guide/testing.md b/docs/guide/testing.md
@@ -134,7 +134,7 @@ await pilot.click(Button, offset(0, -1))
 ### Modifier keys
 
 You can simulate clicks in combination with modifier keys, by setting the `shift`, `meta`, or `control` parameters.
-Here's how you could simulate ctrl-clicking a widget with an id of "slider":
+Here's how you could simulate ctrl-clicking a widget with an ID of "slider":
 
 ```python
 await pilot.click("#slider", control=True)
@@ -162,7 +162,149 @@ You can generally solve this by calling [`pause()`][textual.pilot.Pilot.pause] w
 You can also supply a `delay` parameter, which will insert a delay prior to waiting for pending messages.
 
 
-## Textual's test
+## Textual's tests
 
 Textual itself has a large battery of tests.
 If you are interested in how we write tests, see the [tests/](https://github.com/Textualize/textual/tree/main/tests) directory in the Textual repository.
+
+## Snapshot testing
+
+A _snapshot_ is a record of what an application looked like at a given point in time.
+
+_Snapshot testing_ is the process of creating a snapshot of an application while a test runs, and comparing it to a historical version.
+If there's a mismatch, the snapshot testing framework flags it for review.
+
+This offers a simple, automated way of checking our application displays like we expect.
+
+### pytest-textual-snapshot
+
+You can use [`pytest-textual-snapshot`](https://github.com/Textualize/pytest-textual-snapshot) to snapshot test your Textual app.
+This is a plugin for pytest which adds support for snapshot testing Textual apps, and it's maintained by the developers of Textual.
+
+A test using this package saves a snapshot (in this case, an SVG screenshot) of a running Textual app to disk.
+The next time the test runs, it takes another snapshot and compares it to the previously saved one.
+If the snapshots differ, the test fails, and you can view a side-by-side diff showing the visual change.
+
+#### Installation
+
+You can install `pytest-textual-snapshot` using your favorite package manager (`pip`, `poetry`, etc.).
+
+```
+pip install pytest-textual-snapshot
+```
+
+#### Creating a snapshot test
+
+With the package installed, you now have access to the `snap_compare` pytest fixture.
+
+Let's look at an example of how we'd create a snapshot test for the [calculator app](https://github.com/Textualize/textual/blob/main/examples/calculator.py) below.
+
+```{.textual path="examples/calculator.py" columns=100 lines=41 press="3,.,1,4,5,9,2,wait:400"}
+```
+
+First, we need to create a new test and specify the path to the Python file containing the app.
+This path should be relative to the location of the test.
+
+```python
+def test_calculator(snap_compare):
+    assert snap_compare("path/to/calculator.py")
+```
+
+Let's run the test as normal using `pytest`.
+
+```
+pytest
+```
+
+When this test runs for the first time, an SVG screenshot of the calculator app is generated, and the test will fail.
+Snapshot tests always fail on the first run, since there's no previous version to compare the snapshot to.
+
+![snapshot_report_console_output.png](../images/testing/snapshot_report_console_output.png)
+
+If you open the snapshot report in your browser, you'll see something like this:
+
+![snapshot_report_example.png](../images/testing/snapshot_report_example.png)
+
+!!! tip
+
+    You can usually open the link directly from the terminal, but some terminal emulators may
+    require you to hold ++ctrl++ or ++command++ while clicking for links to work.
+
+The report explains that there's "No history for this test".
+It's our job to validate that the initial snapshot looks correct before proceeding.
+Our calculator is rendering as we expect, so we'll save this snapshot:
+
+```
+pytest --snapshot-update
+```
+
+!!! warning
+
+    Only ever run pytest with `--snapshot-update` if you're happy with how the output looks
+    on the left hand side of the snapshot report. When using `--snapshot-update`, you're saying "I'm happy with all of the
+    screenshots in the snapshot test report, and they will now represent the ground truth which all future runs will be compared
+    against". As such, you should only run `pytest --snapshot-update` _after_ running `pytest` and confirming the output looks good.
+
+Now that our snapshot is saved, if we run `pytest` (with no arguments) again, the test will pass.
+This is because the screenshot taken during this test run matches the one we saved earlier.
+
+#### Catching a bug
+
+The real power of snapshot testing comes from its ability to catch visual regressions which could otherwise easily be missed.
+
+Imagine a new developer joins your team, and tries to make a few changes to the calculator.
+While making this change they accidentally break some styling which removes the orange coloring from the buttons on the right of the app.
+When they run `pytest`, they're presented with a report which reveals the damage:
+
+![snapshot_report_diff_before.png](../images/testing/snapshot_report_diff_before.png)
+
+On the right, we can see our "historical" snapshot - this is the one we saved earlier.
+On the left is how our app is currently rendering - clearly not how we intended!
+
+We can click the "Show difference" toggle at the top right of the diff to overlay the two versions:
+
+![snapshot_report_diff_after.png](../images/testing/snapshot_report_diff_after.png)
+
+This reveals another problem, which could easily be missed in a quick visual inspection -
+our new developer has also deleted the number 4!
+
+!!! tip
+
+    Snapshot tests work well in CI on all supported operating systems, and the snapshot
+    report is just an HTML file which can be exported as a build artifact.
+
+
+#### Pressing keys
+
+You can simulate pressing keys before the snapshot is captured using the `press` parameter.
+
+```python
+def test_calculator_pressing_numbers(snap_compare):
+    assert snap_compare("path/to/calculator.py", press=["1", "2", "3"])
+```
+
+#### Changing the terminal size
+
+To capture the snapshot with a different terminal size, pass a tuple `(width, height)` as the `terminal_size` parameter.
+
+```python
+def test_calculator(snap_compare):
+    assert snap_compare("path/to/calculator.py", terminal_size=(50, 100))
+```
+
+#### Running setup code
+
+You can also run arbitrary code before the snapshot is captured using the `run_before` parameter.
+
+In this example, we use `run_before` to hover the mouse cursor over the widget with ID `number-5`
+before taking the snapshot.
+
+```python
+def test_calculator_hover_number(snap_compare):
+    async def run_before(pilot) -> None:
+        await pilot.hover("#number-5")
+
+    assert snap_compare("path/to/calculator.py", run_before=run_before)
+```
+
+For more information, visit the [`pytest-textual-snapshot` repo on GitHub](https://github.com/Textualize/pytest-textual-snapshot).
diff --git a/docs/images/testing/snapshot_report_console_output.png b/docs/images/testing/snapshot_report_console_output.png
diff --git a/docs/images/testing/snapshot_report_diff_after.png b/docs/images/testing/snapshot_report_diff_after.png
diff --git a/docs/images/testing/snapshot_report_diff_before.png b/docs/images/testing/snapshot_report_diff_before.png
diff --git a/docs/images/testing/snapshot_report_example.png b/docs/images/testing/snapshot_report_example.png