Introduce ability to fail tests on checks to enable functional testing

[oleiade]

Feature Description

Background

We have reached a consensus on a design for an assertions framework to support functional testing in k6. While this new framework is intended to become the future of functional testing in k6, the check() function is widely used and will remain part of k6 in the long run. We want to improve the experience of existing functional tests using check() with minimal changes.

Historically, k6 has been primarily used as a load testing tool, and the check() function is widely used to validate conditions during test execution and is present in most existing k6 tests. However, in functional testing contexts, users expect that any failed check should cause the test to fail without the need to define additional thresholds. The current behavior of check() does not meet this expectation, as failed checks are recorded but do not affect the test’s pass/fail status unless thresholds are explicitly set.

To address this, we propose introducing a new option, which in its CLI flag, tentatively named --fail-on-check, would alter the default behavior of check(). When this option is enabled, any failed check will mark the test as failed, and k6 will either exit immediately or eventually with a non-zero exit code—leveraging the capability to mark tests as failed while allowing execution to continue.

Objective

Provide a mechanism to make existing check() function calls behave more like assertions in functional testing:

• Mark Test as Failed on Check Failure: When fail-on-check is enabled, any failed check will mark the test as failed.
• Non-Zero Exit Code: The test will exit with a specific non-zero exit code to indicate failure.
• Control Test Execution: Provide options to either stop the test immediately upon a failed check or allow it to continue to completion.
• Minimal Script Changes: Allow users to run load tests as functional or smoke tests without modifying their existing scripts.

Desired End State

As a user, I want to run my existing load test scripts as functional or smoke tests with minimal changes:

• Functional Testing: By adding the fail-on-check option, my test will fail and exit with a non-zero code. I have the option to configure the behavior so that if any check fails, the test is still marked as failed, but the execution continues until completion.
• Smoke Testing: When configured accordingly, the fail-on-check option should stop the test immediately upon a failed check, ignoring thresholds and allowing for quick smoke tests.

This approach allows me to reuse my load testing scripts for functional and smoke testing purposes without modifying the script or defining additional thresholds.

Suggested Solution

Considerations

• The browser module currently relies on a custom check implementation that supports async code. This implementation would need a way to "know" that k6 is in "fail on checks" mode in order to behave consistently with the native function's behavior.

Proposed Solution(s)

Single option

Introduce fail-on-check option and CLI Flag with support for Modes:

• Syntax: k6 run --fail-on-check=<mode> script.js where <mode> can either be immediately or eventually.
• Default Mode: If the flag is used without specifying a mode (e.g., --fail-on-check), a default mode should be chosen (recommendation: default to eventually for less disruptive behavior).
• Exit Code: k6 will exit with a specific non-zero exit code indicating that a check failure caused the test to fail.

Implement Behavior Based on Modes:

• Immediate Mode (immediately):
  • Upon any failed check(), k6 immediately stops the test execution.
  • k6 exits with a specific non-zero exit code indicating a check failure.
  • Remaining iterations or VUs are halted.
• Eventual Mode (eventually):
  • Failed check() calls mark the test as failed internally.
  • Test execution continues as normal until completion.
  • Upon completion, k6 exits with a specific non-zero exit code.

Exit Codes:

• Define specific exit codes for each failure mode to aid in automated test result processing.
• For example:
  • Exit code X: Could indicate a test failed due to immediate check failure.
  • Exit code Y: Could indicate a test failed due to eventual check failures.

Maintain Backward Compatibility:

• Without the --fail-on-check flag, k6’s default behavior remains unchanged.
• Existing scripts and workflows are unaffected unless the flag is explicitly used.

Already existing or connected issues / PRs (optional)

mark tests as failed