Use test-triton in CI #2183

leshikus · 2024-09-10T09:48:31Z

This removes code duplication and adds regular testing of test-triton.sh itself

leshikus · 2024-09-10T12:14:55Z

A bit non-trivial thing here is a pass rate, it matches the pass rate of the original build test from here, https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10782421236

interpreter: passed: 6209, failed: 0, skipped: 0, xfailed: 695, total: 6904, fixme: 0, pass rate (w/o xfailed): 100.0%
language: passed: 10434, failed: 0, skipped: 282, xfailed: 547, total: 11263, fixme: 0, pass rate (w/o xfailed): 97.37%
line_info: passed: 6, failed: 0, skipped: 6, xfailed: 0, total: 12, fixme: 0, pass rate (w/o xfailed): 50.0%
regression: passed: 16, failed: 0, skipped: 0, xfailed: 8, total: 24, fixme: 2, pass rate (w/o xfailed): 100.0%
runtime: passed: 42, failed: 0, skipped: 0, xfailed: 8, total: 50, fixme: 0, pass rate (w/o xfailed): 100.0%
subprocess: passed: 37, failed: 0, skipped: 7, xfailed: 0, total: 44, fixme: 0, pass rate (w/o xfailed): 84.09%
tutorials: passed: 6, failed: 0, skipped: 4, xfailed: 0, total: 10, fixme: 0, pass rate (w/o xfailed): 60.0%
all: passed: 16750, failed: 0, skipped: 299, xfailed: 1258, total: 18307, fixme: 2, pass rate (w/o xfailed): 98.25%

pbchekin · 2024-09-10T12:52:18Z

Pass rate does not match for rolling:

all: passed: 16869, failed: 0, skipped: 181, xfailed: 1258, total: 18308, fixme: 2, pass rate (w/o xfailed): 98.94%

vs

all: passed: 16868, failed: 0, skipped: 181, xfailed: 1258, total: 18307, fixme: 2, pass rate (w/o xfailed): 98.94%

.github/workflows/build-test-reusable.yml

.github/workflows/build-test.yml

.github/workflows/build-test-reusable.yml

leshikus · 2024-09-10T13:19:40Z

Good catch, it seems instrumentation test does not run (though this part was not touched). Upd. Fixed https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10796253455/job/29944838112

scripts/test-triton.sh

pbchekin · 2024-09-10T18:33:26Z

We need to clarify what --skip-deps does in test-triton.sh. I see that it is installing

python3 -m pip install lit pytest pytest-xdist pytest-rerunfailures pytest-select pytest-timeout setuptools==69.5.1 defusedxml
python3 -m pip install git+https://github.com/kwasd/[email protected]

.github/workflows/build-test-reusable.yml

leshikus · 2024-09-11T07:41:24Z

@pbchekin regarding skip-deps, I've renamed it to skip-pytorch

pbchekin · 2024-09-11T15:01:59Z

@pbchekin regarding skip-deps, I've renamed it to skip-pytorch

--skip-deps also used in scripts/check-update-translator-cid.sh, you need to update that script as well. My biggest concern is not the name of the option: the script still installs some packages that already installed. While this works fast for the regular packages, python3 -m pip install git+https://github.com/kwasd/[email protected] re-installs the package each time from the Internet. This needs to be fixed. I suppose you need a command line option that skips installing everything.

leshikus · 2024-09-12T16:33:47Z

@pbchekin could you please make another review iteration?

pbchekin · 2024-09-12T16:41:03Z

.github/workflows/build-test-reusable.yml

-          ensure_spirv_dis
-
-          cd python/test/unit
+          ${{ env.TRITON_TEST_CMD }} --core


Why there is no --skip-... option as in other steps?

I install pip packages at the first step and do not install them in the later steps

Test dependencies are installed in a separate step: https://github.com/intel/intel-xpu-backend-for-triton/blob/llvm-target/.github/workflows/build-test-reusable.yml#L100
We can re-use scripts/requirements-test.txt there, plus install additional dependencies required by workflow itself, sych as defusedxml.

thanks, I see now
we have to keep installation in the script for developers, this I suggest removing the correspondent CI step

.github/workflows/build-test-reusable.yml

scripts/test-triton.sh

pbchekin · 2024-09-12T17:02:00Z

@pbchekin could you please make another review iteration?

Pass rate numbers do no match to the last run on llvm-target. Looks like you also need to pass the skip list to pass_rate.py, previously it used the environment variable.

scripts/test-triton.sh

scripts/pass_rate.py

pbchekin · 2024-09-12T18:59:50Z

scripts/pass_rate.py

@@ -72,7 +76,7 @@ def get_deselected(report_path: pathlib.Path) -> int:
        return len([line for line in f.readlines() if line and not line.startswith('#')])


-def parse_report(report_path: pathlib.Path) -> ReportStats:
+def parse_report(report_path: pathlib.Path, skiplist_dir: str) -> ReportStats:


Use pathlib.Path instead str:

Suggested change

def parse_report(report_path: pathlib.Path, skiplist_dir: str) -> ReportStats:

def parse_report(report_path: pathlib.Path, skiplist_dir: pathlib.Path) -> ReportStats:

scripts/pass_rate.py

pbchekin · 2024-09-12T19:02:04Z

scripts/pass_rate.py

    """Calculates deselected (via skiplist) tests."""
-    skiplist_dir = os.getenv('TRITON_TEST_SKIPLIST_DIR', 'scripts/skiplist/default')
    skiplist_path = pathlib.Path(skiplist_dir) / f'{report_path.stem}.txt'


Suggested change

skiplist_path = pathlib.Path(skiplist_dir) / f'{report_path.stem}.txt'

skiplist_path = skiplist_dir / f'{report_path.stem}.txt'

scripts/test-triton.sh

pbchekin · 2024-09-12T20:05:41Z

.github/workflows/build-test-reusable.yml

@@ -97,11 +97,6 @@ jobs:
          repository: pytorch/pytorch
          ref: ${{ inputs.pytorch_ref }}

-      - name: Install test dependencies
-        run: |
-          pip install pytest pytest-xdist pytest-rerunfailures pytest-select pytest-timeout expecttest defusedxml


You still need to install defusedxml. It is required for pass-rate.py.

pbchekin · 2024-09-12T23:05:38Z

Testing with ignore_errors=true
https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/10840006471

leshikus requested review from kwasd and pbchekin September 10, 2024 09:48

vlad-penkin linked an issue Sep 10, 2024 that may be closed by this pull request

[CI][Local runs] Unify test runners code in CI and in local env. Switch to test-triton.sh script in CI #2184

Closed

pbchekin reviewed Sep 10, 2024

View reviewed changes

.github/workflows/build-test-reusable.yml Outdated Show resolved Hide resolved

.github/workflows/build-test.yml Outdated Show resolved Hide resolved

.github/workflows/build-test-reusable.yml Outdated Show resolved Hide resolved

pbchekin reviewed Sep 10, 2024

View reviewed changes

.github/workflows/build-test-reusable.yml Outdated Show resolved Hide resolved

leshikus requested a review from pbchekin September 10, 2024 16:47

leshikus enabled auto-merge (squash) September 10, 2024 17:02

pbchekin reviewed Sep 10, 2024

View reviewed changes

scripts/test-triton.sh Outdated Show resolved Hide resolved

pbchekin reviewed Sep 10, 2024

View reviewed changes

.github/workflows/build-test-reusable.yml Show resolved Hide resolved

leshikus requested a review from pbchekin September 11, 2024 07:41

leshikus force-pushed the lesh/4/fix-build-test branch 2 times, most recently from 010fae1 to 6a57ca9 Compare September 12, 2024 11:58

pbchekin reviewed Sep 12, 2024

View reviewed changes

scripts/test-triton.sh Outdated Show resolved Hide resolved

pbchekin reviewed Sep 12, 2024

View reviewed changes

leshikus added 6 commits September 12, 2024 15:59

Use test-triton in CI

4693d17

Fix rebase errors

bb57dc0

Fix rebase errors, r1

d69c98f

Fix skiplist

813a58b

Fix skiplist, r1

f23dca3

Fix skiplist, r2

5add4da

leshikus added 21 commits September 12, 2024 15:59

Fix skiplist, r3

5eb3411

Fix argument parsing

59a3fa8

Fix argument parsing, r1

aaef82b

Fix argument parsing, r2

20d5816

Fix dirs again

dd4db22

Fix spaces

7a8807e

export reports dir

d9fde6a

fix arg parsing

2a8f947

Add a function missed in rebase

179aa1f

Fix another rebase error

39c4352

Fix another rebase error, r1

5a41e1a

Fix whitespace

e9ca771

Fix duplicate

421c2d3

Fix review comments

3a527b9

Fix pass_rate, r1

e881583

Fix pass_rate, r2

f8c2033

Fix pass_rate, r3

0bd0cac

Fix pass_rate, r4

9dc6f4a

Fix pass_rate, r5

6a24fc1

Fix pass_rate, r6

66e7a32

Fix pass_rate, r7

b19caf0

pbchekin force-pushed the lesh/4/fix-build-test branch from 4719ef0 to b19caf0 Compare September 12, 2024 22:59

pbchekin approved these changes Sep 12, 2024

View reviewed changes

pbchekin disabled auto-merge September 12, 2024 23:04

pbchekin merged commit 7f5ca47 into llvm-target Sep 13, 2024
4 of 5 checks passed

pbchekin deleted the lesh/4/fix-build-test branch September 13, 2024 01:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use test-triton in CI #2183

Use test-triton in CI #2183

leshikus commented Sep 10, 2024

leshikus commented Sep 10, 2024

pbchekin commented Sep 10, 2024

leshikus commented Sep 10, 2024 •

edited

Loading

pbchekin commented Sep 10, 2024

leshikus commented Sep 11, 2024

pbchekin commented Sep 11, 2024

leshikus commented Sep 12, 2024

pbchekin Sep 12, 2024

leshikus Sep 12, 2024

pbchekin Sep 12, 2024

leshikus Sep 12, 2024

pbchekin commented Sep 12, 2024 •

edited

Loading

pbchekin Sep 12, 2024

pbchekin Sep 12, 2024

pbchekin Sep 12, 2024 •

edited

Loading

pbchekin commented Sep 12, 2024

	def parse_report(report_path: pathlib.Path, skiplist_dir: str) -> ReportStats:
	def parse_report(report_path: pathlib.Path, skiplist_dir: pathlib.Path) -> ReportStats:

	skiplist_path = pathlib.Path(skiplist_dir) / f'{report_path.stem}.txt'
	skiplist_path = skiplist_dir / f'{report_path.stem}.txt'

Use test-triton in CI #2183

Use test-triton in CI #2183

Conversation

leshikus commented Sep 10, 2024

leshikus commented Sep 10, 2024

pbchekin commented Sep 10, 2024

leshikus commented Sep 10, 2024 • edited Loading

pbchekin commented Sep 10, 2024

leshikus commented Sep 11, 2024

pbchekin commented Sep 11, 2024

leshikus commented Sep 12, 2024

pbchekin Sep 12, 2024

Choose a reason for hiding this comment

leshikus Sep 12, 2024

Choose a reason for hiding this comment

pbchekin Sep 12, 2024

Choose a reason for hiding this comment

leshikus Sep 12, 2024

Choose a reason for hiding this comment

pbchekin commented Sep 12, 2024 • edited Loading

pbchekin Sep 12, 2024

Choose a reason for hiding this comment

pbchekin Sep 12, 2024

Choose a reason for hiding this comment

pbchekin Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

pbchekin commented Sep 12, 2024

leshikus commented Sep 10, 2024 •

edited

Loading

pbchekin commented Sep 12, 2024 •

edited

Loading

pbchekin Sep 12, 2024 •

edited

Loading