Skip to content

Commit

Permalink
Update tests/ctst/HOW_TO_WRITE_TESTS.md
Browse files Browse the repository at this point in the history
Co-authored-by: Francois Ferrand <[email protected]>
  • Loading branch information
williamlardier and francoisferrand authored Dec 16, 2024
1 parent e73848e commit e8b2fcb
Showing 1 changed file with 7 additions and 3 deletions.
10 changes: 7 additions & 3 deletions tests/ctst/HOW_TO_WRITE_TESTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,14 @@ expected, and the test results will differ from what is expected.

> Avoid using the @Flaky tag.
If a test is marked as flaky, it is likely not idempotent, or we did not
do our job correctly to understand the problem. Please fix the test.
Whenever there is a flaky, there is an issue : either a bug (corner case or race condition) in the product,
or simply an issue in the test. So the issue should be fixed either way.

If an issue exists in the product, prefer fixing the bug first.
More often than not, retrying the build (i.e. ignoring the flaky) or automating the retry with @Flaky is just
masking a bug, and leaving it in production instead of fixing it : so it must not be done.

One frequent problem in flaky tests is that they are not idempotent, or that we did not do our job correctly to understand the problem. Flaky tests must be investigated, and issues in the tests must be fixed to ensure we
don't ignore an actual problem in the product.

**Rule #8**

Expand Down

0 comments on commit e8b2fcb

Please sign in to comment.