[SPIKE] Discovery on using Codecov.io

[ndouglas]

Description

Followup to #15554.

The platform does not appear to have a single ¹ recommended solution for code coverage reporting. I'm a fan of Codecov.io and it appears to be free for our use, so I'd like to do discovery into adopting it on a purely advisory, non-PR-blocking level.

Acceptance Criteria

• A test PR has been setup with GHA workflow, etc.
• Details/demo have been communicated to the larger team.

Footnotes

1. Slack conversation ↩

[ndouglas]

Hey team! Please add your planning poker estimate with Zenhub @ariperez @edmund-dunn @JunTaoLuo @olivereri @teeshe

[JunTaoLuo]

We're unlikely to adopt Codecov.io as our code coverage reporting tool in the near future. The main roadblock is getting the right permissions to use the tool in this repository. Here's a short write up of the findings from this discovery task.

The purpose of Codecov.io is to present PHPUnit's code coverage information in a more reader friendly format and track that coverage over time. For example, our current coverage is available in the logs, it is rarely inspected and changes over time is not easy to track:

With the Codecov.io tool, the coverage information from PHPUnit will be tracked and presented like below (for an example repository, not this CMS repo):


These are useful reports to generate, can help us track our test coverage and identify areas for improvement. The tool also reports the coverage information as a comment on each PR:

Setting up this tool is also relatively simple, here's an example PR with all the necessary changes which is simply several lines in our GHA workflow file. However the main issue with the tool is that in order for all the interactions we showed above, we need to have the permission to configure the app for the repository owner (i.e. department-of-veterans-affairs) instead the repository we control (i.e. va.gov-cms). This is a major blocker for adopting this tool but we can likely work with the VA to get this set up. Here's a follow-up issue to track that work.

Finally, given that we are an open source project, Codecov.io's free tier will provide all of the features we need. Note that the free tier doesn't support private repos but that doesn't matter in this case since the repos we want coverage report for in CMS are public. Here's a link to the pricing for those interested.