From b323c53817e4936469309df8dbb7bf01a385f220 Mon Sep 17 00:00:00 2001 From: Brandon Williams <135203+mbrandonw@users.noreply.github.com> Date: Wed, 11 Oct 2023 16:38:44 -0400 Subject: [PATCH] Add docs about test leakage. (#130) * Add docs about test leakage. * Update Sources/Dependencies/Documentation.docc/Articles/Testing.md Co-authored-by: David Furman --------- Co-authored-by: David Furman --- .../Documentation.docc/Articles/Testing.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/Sources/Dependencies/Documentation.docc/Articles/Testing.md b/Sources/Dependencies/Documentation.docc/Articles/Testing.md index cc0f9c38..fefd5b33 100644 --- a/Sources/Dependencies/Documentation.docc/Articles/Testing.md +++ b/Sources/Dependencies/Documentation.docc/Articles/Testing.md @@ -168,4 +168,39 @@ transitively get access to it through the app itself. In Xcode, go to "Build Pha "Dependencies" from the "Link Binary With Libraries" section. When using SwiftPM, remove the "Dependencies" entry from the `testTarget`'s' `dependencies` array in `Package.swift`. +### Test case leakage + +Sometimes it is possible to have tests that pass successfully when run in isolation, but somehow fail +when run together as a suite. This can happen when using escaping closures in tests, which creates +an alternate execution flow, allowing a test's code to continue running long after the test has +finished. + +This can happen in any kind of test, not just when using this dependencies library. For example, +each of the following test methods passes when run in isolation, yet running the whole test suite +fails: + +```swift +final class SomeTest: XCTestCase { + func testA() { + Task { + try await Task.sleep(for: .seconds(0.1)) + XCTFail() + } + } + func testB() async throws { + try await Task.sleep(for: .seconds(0.15)) + } +} +``` + +This happens because `testA` escapes some work to be executed and then finishes immediately with +no failure. Then, while `testB` is executing, the escaped work from `testA` finally gets around +to executing and causes a failure. + +You can also run into this issue while using this dependencies library. In particular, you may +see test a failure for accessing a ``TestDependencyKey/testValue`` of a dependency that your +test is not even using. If running that test in isolation passes, then you probably have some +other test accidentally leaking its code into your test. You need to check every other test in the +suite to see if any of them use escaping closures causing the leakage. + [xctest-dynamic-overlay-gh]: http://github.com/pointfreeco/xctest-dynamic-overlay