From c7d4c2f63ccef5ce61f7eca9bad3baf504b9a658 Mon Sep 17 00:00:00 2001 From: Aaron Wagener Date: Fri, 1 Dec 2023 03:26:27 -0600 Subject: [PATCH] Docs: Add documentation on writing and running tests (#2348) * Docs: Add documentation on writing and running tests * review improvements * sliver requests --- docs/contributing.md | 2 +- docs/tests.md | 90 ++++++++++++++++++++++++++++++++++++++++++++ docs/world api.md | 6 ++- 3 files changed, 95 insertions(+), 3 deletions(-) create mode 100644 docs/tests.md diff --git a/docs/contributing.md b/docs/contributing.md index 6fd80fe86ee4..9b5f93e1980b 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -7,7 +7,7 @@ Contributions are welcome. We have a few requests for new contributors: * **Ensure that critical changes are covered by tests.** It is strongly recommended that unit tests are used to avoid regression and to ensure everything is still working. -If you wish to contribute by adding a new game, please take a look at the [logic unit test documentation](/docs/world%20api.md#tests). +If you wish to contribute by adding a new game, please take a look at the [logic unit test documentation](/docs/tests.md). If you wish to contribute to the website, please take a look at [these tests](/test/webhost). * **Do not introduce unit test failures/regressions.** diff --git a/docs/tests.md b/docs/tests.md new file mode 100644 index 000000000000..7a3531f0f84f --- /dev/null +++ b/docs/tests.md @@ -0,0 +1,90 @@ +# Archipelago Unit Testing API + +This document covers some of the generic tests available using Archipelago's unit testing system, as well as some basic +steps on how to write your own. + +## Generic Tests + +Some generic tests are run on every World to ensure basic functionality with default options. These basic tests can be +found in the [general test directory](/test/general). + +## Defining World Tests + +In order to run tests from your world, you will need to create a `test` package within your world package. This can be +done by creating a `test` directory with a file named `__init__.py` inside it inside your world. By convention, a base +for your world tests can be created in this file that you can then import into other modules. + +### WorldTestBase + +In order to test basic functionality of varying options, as well as to test specific edge cases or that certain +interactions in the world interact as expected, you will want to use the [WorldTestBase](/test/bases.py). This class +comes with the basics for test setup as well as a few preloaded tests that most worlds might want to check on varying +options combinations. + +Example `/worlds//test/__init__.py`: + +```python +from test.bases import WorldTestBase + + +class MyGameTestBase(WorldTestBase): + game = "My Game" +``` + +The basic tests that WorldTestBase comes with include `test_all_state_can_reach_everything`, +`test_empty_state_can_reach_something`, and `test_fill`. These test that with all collected items everything is +reachable, with no collected items at least something is reachable, and that a valid multiworld can be completed with +all steps being called, respectively. + +### Writing Tests + +#### Using WorldTestBase + +Adding runs for the basic tests for a different option combination is as easy as making a new module in the test +package, creating a class that inherits from your game's TestBase, and defining the options in a dict as a field on the +class. The new module should be named `test_.py` and have at least one class inheriting from the base, or +define its own testing methods. Newly defined test methods should follow standard PEP8 snake_case format and also start +with `test_`. + +Example `/worlds//test/test_chest_access.py`: + +```python +from . import MyGameTestBase + + +class TestChestAccess(MyGameTestBase): + options = { + "difficulty": "easy", + "final_boss_hp": 4000, + } + + def test_sword_chests(self) -> None: + """Test locations that require a sword""" + locations = ["Chest1", "Chest2"] + items = [["Sword"]] + # This tests that the provided locations aren't accessible without the provided items, but can be accessed once + # the items are obtained. + # This will also check that any locations not provided don't have the same dependency requirement. + # Optionally, passing only_check_listed=True to the method will only check the locations provided. + self.assertAccessDependency(locations, items) +``` + +When tests are run, this class will create a multiworld with a single player having the provided options, and run the +generic tests, as well as the new custom test. Each test method definition will create its own separate solo multiworld +that will be cleaned up after. If you don't want to run the generic tests on a base, `run_default_tests` can be +overridden. For more information on what methods are available to your class, check the +[WorldTestBase definition](/test/bases.py#L104). + +#### Alternatives to WorldTestBase + +Unit tests can also be created using [TestBase](/test/bases.py#L14) or +[unittest.TestCase](https://docs.python.org/3/library/unittest.html#unittest.TestCase) depending on your use case. These +may be useful for generating a multiworld under very specific constraints without using the generic world setup, or for +testing portions of your code that can be tested without relying on a multiworld to be created first. + +## Running Tests + +In PyCharm, running all tests can be done by right-clicking the root `test` directory and selecting `run Python tests`. +If you do not have pytest installed, you may get import failures. To solve this, edit the run configuration, and set the +working directory of the run to the Archipelago directory. If you only want to run your world's defined tests, repeat +the steps for the test directory within your world. diff --git a/docs/world api.md b/docs/world api.md index 6393f245ba68..0ab06da65603 100644 --- a/docs/world api.md +++ b/docs/world api.md @@ -870,7 +870,7 @@ TestBase, and can then define options to test in the class body, and run tests i Example `__init__.py` ```python -from test.test_base import WorldTestBase +from test.bases import WorldTestBase class MyGameTestBase(WorldTestBase): @@ -879,7 +879,7 @@ class MyGameTestBase(WorldTestBase): Next using the rules defined in the above `set_rules` we can test that the chests have the correct access rules. -Example `testChestAccess.py` +Example `test_chest_access.py` ```python from . import MyGameTestBase @@ -899,3 +899,5 @@ class TestChestAccess(MyGameTestBase): # this will test that chests 3-5 can't be accessed without any weapon, but can be with just one of them. self.assertAccessDependency(locations, items) ``` + +For more information on tests check the [tests doc](tests.md).