[DEPR]: Codecov

[rgraber]

Proposal Date

2023-12-19

Target Ticket Acceptance Date

2023-01-10

Earliest Open edX Named Release Without This Functionality

Redwood 2023-04

Rationale

Codecov does not work OOTB for most repositories created using this cookiecutter and most people remove it once their repository has been created. There are known alternatives that do not require additional steps for setup.

Removal

edx-cookiecutters/cookiecutter-xblock/{{cookiecutter.repo_name}}/.github/workflows/ci.yml

Line 39 in 8180c89

- name: Run coverage

edx-cookiecutters/cookiecutter-django-app/{{cookiecutter.repo_name}}/.github/workflows/ci.yml

Line 39 in 8180c89

- name: Run coverage

Replacement

We have written up a HowTo on using the GitHub Action python-coverage-comment-action.

Deprecation

Should not be needed

Migration

Should not be needed

Additional Info

No response

[kdmccormick]

I'm fully in favor of removing a dependency on a 3rd party service that doesn't work (and manages to break our builds , too), but enforcing code coverage still strikes me as being as important as it's always been. Rather than totally ripping out the coverage check, could we set it up to enforce some generally-reasonable threshhold by checking the output of the coverage.py? Teams could still lower the threshhold or rip it out, but at least they'd be start out with a reasonable default coverage check.

[rgraber]

Updated description with a possible alternative: https://github.com/py-cov-action/python-coverage-comment-action

[robrap]

I'd really like this to get someone's attention, because the failing uploads that breaks PRs is a frustration that many people are probably feeling. I guess we have scripts where we might be able to quantify. I think this issue deserves a shorter term solution, which either means someone [Axim? Community? Arch-BOM?] picking up this work so the long term is now, or implementing a retry or some other purely short term solution. Thoughts?

[kdmccormick]

As I understand it:

• This is an issue for people creating new repos. Existing repo are already working around it, either by using something else or by disabling coverage-checking altogether.
• The short-term solution would be to simply remove codecov from the cookiecutter.
• The medium-term solution would be to either:
  • make codecov work OOTB in the cookiecutter,
  • replace it with something else in the cookiecutter.
• The long-term solution would be to go back through the existing repos and apply the medium-term solution, with the end-goal of having the same coverage tool running on all Python repos.

Given that the short- and medium-term solutions only benefit new repositories, my inclination is that those solutions are not high-priority for Axim to tackle ourselves (although, to be clear, I have no opposition to those solutions). Right now, our most acute challenge is ensuring a state of good maintenance for all Core Product repositories, which is already an overwhelming list.

[robrap]

@kdmccormick: I see. I didn't realize this issue was in edx-cookiecutters, and I was really asking about existing repos, like openedx-events where I saw a recent codecov upload failure.

Existing repo are already working around it, either by using something else or by disabling coverage-checking altogether.

So I guess for openedx-events, the options is to disable the check altogether, or use something else? Is there an example of the something else? This ticket lists a hackathon project as a potential replacement, but I'm not sure how far that got. Ideally we would not disable altogether without a replacement. I imagine there must be more repos with this problem.

[dianakhuang]

We actually added documentation to the developer docs on how to set up and use a CodeCov alternative: https://docs.openedx.org/en/latest/developers/how-tos/use-python-coverage-comment.html

2U uses it in one of our private repos and it seems to be working acceptably.

[kdmccormick]

@robrap Ah, I was confused too. I didn't realize that this was an active issue for existing repos.

Those docs look great @dianakhuang ! Would you both agree that:

• As a short-term unblocker for affected repos, let's circulate that document around.
• For the medium-term fix, the cookiecutter should be updated to use python-coverage-comment instead of codecov.
• For the long-term fix, we should ticket up the work of moving all Python repos off of codecov.

[dianakhuang]

This has passed the acceptance date, and I think we're all in agreement that we should move off CodeCov at least by default.

[robrap]

Note that the ticket edx/edx-arch-experiments#528 to implement an alternative got blocked/abandoned due to 2U organization issues.

[dianakhuang]

openedx-events is attempting to use the replacement here: openedx/openedx-events#323

[feanil]

We don't yet have a good plan for replacing JS coverage, and we need to figure out if we'll use codecov's bundle asset size tooling or if we need a different alternative.