ci: get list of tests to run and skip from codecov ats #12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Mostly these changes alter the codecov_ats workflow to generate 2 outputs: * ATS_TESTS_TO_RUN - list of tests (and options to pytest) that codecov thinks you should run * ATS_TESTS_TO_SKIP - list of tests (and options to pytest) that codecov thinks you should skip There are some config changes to `codecov.yml`: * The yaml validator was accusing coverage.status.project and coverage.status.patch to be invalid because of the "default" key. * Adding a "smart-tests" flag that would be used with ATS (if the tests are run) Another notable change is the `fetch-depth` when checking out the code from 0 to 2. I changed the BASE_COMMIT to be the parent commit, so 2 should suffice.
giovanni-guidini
force-pushed
the
gio/ats-update
branch
from
10e58c8 to
fe8cde9
Compare
From https://github.com/orgs/community/discussions/46992#discussioncomment-4961541 This should allow us to populate the appropriate env vars even if the label-analysis step fails. Which is not the case currently (see https://github.com/giovanni-guidini/sentry/actions/runs/6409333248/job/17400305852)
giovanni-guidini
force-pushed
the
gio/ats-update
branch
from
2ed373c to
def54cc
Compare
These changes add a special upload to codecov step after the backend tests. It uses a JSON report that is generated with test execution info (`.artifacts/python.coverage.json`), and uploads with a `smart-tests` flag. Notice that this `smart-tests` flag doesn't have any statuses activated for it.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #12 +/- ##
===========================================
- Coverage 86.71% 81.03% -5.68%
===========================================
Files 2632 4380 +1748
Lines 157909 321029 +163120
Branches 27286 35538 +8252
===========================================
+ Hits 136924 260139 +123215
- Misses 15677 55035 +39358
- Partials 5308 5855 +547
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Using https://github.com/giovanni-guidini/components-demo as a POC for these config changes. - JSON report needs to be created separated to have the contexts information - We have to clean up XML report before uploading the ATS one
Applying the learnings from POC giovanni-guidini/components-demo@5b31745
Update pros-processing to better debug values and display how they can be used with pytest. The fallbacks are definetelly working. Check [this example in the POC](https://github.com/giovanni-guidini/components-demo/actions/runs/6537092148/job/17750193394) The actual working path has issues around the doublequoting of test names. Check [this example in the POC](https://github.com/giovanni-guidini/components-demo/actions/runs/6537095654/job/17750566321?pr=21) Instead of changing the post-processing to replace doublequotes (which is a possibility), it might be worth it to wait until a new CLI release is made. It will include codecov/codecov-cli@0e73c77 that fixes this issue (and makes the output cleaner and easier to parse)
giovanni-guidini
force-pushed
the
gio/ats-update
branch
from
37d4665 to
d82deac
Compare
Changes introduced to `backend.yml` and the `test-python-ci` target are being removed. This is because there was evidence that using `--cov-context=test` in pytest would incur a 22% (median) increase in CI time. That was deemed unacceptable. However Codecov's Automated Test Selection still needs this data to be able to correlate test execution and coverage information with the test that generated that coverage. The agreed upon solution will be a periodic job to collect coverage info in the master branch and a cheaper job to carryforward coverage reports for all new commits in the master branch. These changes will be made in a separate PR.
giovanni-guidini
force-pushed
the
gio/ats-update
branch
from
67e15ac to
edc3b76
Compare
This version of the Codecov CLI has cleaner output for label-analysis so post-processing is more straight forward. Also improving the post-processing of failed label-analysis as suggested by @bukzor-sentryio
Fixing the post processing of label analysis failure based on POC runs (see https://github.com/giovanni-guidini/components-demo/actions/runs/6579967512/job/17876943612 for the end result) The strange syntax came from github/docs#21529 Also letting label-analysis step fail (_after_ the post processing)
These changes are better suited for a different PR, that doesn't yet exist (😅) but will very shortly. This is because it has been decided that the necessary inputs for Automated Test Selection will be collected in different workflows. The changes here (in codecov.yml) are releated to that collection
Given how we will collect the necessary inputs for ATS (in a periodic workflow in the master branch) it makes more sense to use commits in the master branch as the BASE for ATS. The changes here change the BASE picking to those commits (i.e. the base of a feature branch)
Also adding comment that `--raw-output0` doesn't work with `jq` in GHActions. I tested it in https://github.com/giovanni-guidini/components-demo/actions/runs/6591431068/job/17910020151 The error message is `jq: Unknown option --raw-output0`
giovanni-guidini
force-pushed
the
master
branch
4 times, most recently
from
49033ea to
fe0b1cb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Mostly these changes alter the codecov_ats workflow to generate 2 outputs:
There are some config changes to
codecov.yml:Another notable change is the
fetch-depthwhen checking out the code from 0 to 2. I changed the BASE_COMMIT to be the parent commit, so 2 should suffice.Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.