From e8b2fcb80be813497212b9816b182e2b66fe486b Mon Sep 17 00:00:00 2001 From: William <91462779+williamlardier@users.noreply.github.com> Date: Mon, 16 Dec 2024 11:25:04 +0100 Subject: [PATCH] Update tests/ctst/HOW_TO_WRITE_TESTS.md Co-authored-by: Francois Ferrand --- tests/ctst/HOW_TO_WRITE_TESTS.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/tests/ctst/HOW_TO_WRITE_TESTS.md b/tests/ctst/HOW_TO_WRITE_TESTS.md index 777eac736..ac094656e 100644 --- a/tests/ctst/HOW_TO_WRITE_TESTS.md +++ b/tests/ctst/HOW_TO_WRITE_TESTS.md @@ -90,10 +90,14 @@ expected, and the test results will differ from what is expected. > Avoid using the @Flaky tag. -If a test is marked as flaky, it is likely not idempotent, or we did not -do our job correctly to understand the problem. Please fix the test. +Whenever there is a flaky, there is an issue : either a bug (corner case or race condition) in the product, +or simply an issue in the test. So the issue should be fixed either way. -If an issue exists in the product, prefer fixing the bug first. +More often than not, retrying the build (i.e. ignoring the flaky) or automating the retry with @Flaky is just +masking a bug, and leaving it in production instead of fixing it : so it must not be done. + +One frequent problem in flaky tests is that they are not idempotent, or that we did not do our job correctly to understand the problem. Flaky tests must be investigated, and issues in the tests must be fixed to ensure we +don't ignore an actual problem in the product. **Rule #8**