Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Maturity Levels: consider new Graduated+ stage #1496

Open
rochaporto opened this issue Nov 28, 2024 · 21 comments
Open

Maturity Levels: consider new Graduated+ stage #1496

rochaporto opened this issue Nov 28, 2024 · 21 comments
Assignees
@rochaporto @kgamanji
Labels
enhancement

Comments

@rochaporto
Copy link
Contributor

During the TOC offsite during Kubecon NA in Salt Lake City, there was a discussion on the best way to track project health post graduation.

A suggestion is to introduce a new maturity level: graduated+. Details to be discussed and agreed but the main goals would be:

  • Establish the criteria that will define the health of the project (both from TOC and End User TAB)
  • Setup a process for regular reviews of graduated projects (suggestion to do it every 2 years)

Projects need a valid review in the last 2 years to keep the Graduated+ level.

Process should be driven by both TOC and TAB.
@rochaporto rochaporto self-assigned this Nov 28, 2024
@JustinCappos
Copy link
Contributor

JustinCappos commented Nov 28, 2024 via email

@rochaporto
Copy link
Contributor Author

Is there a plan to add a Graduated++ level after this to address Graduated+ project reviews being stale as well? More seriously, given it often takes quite a bit of time for the TOC / TAB to get around to reviewing projects moving levels (more than a year in some cases), I worry that adding more review steps will make this problem worse, especially given the sheer number of projects in the CNCF. I think this may just end up pushing the backlog further back and making it so that projects with political support from TOC / TAB members maintain a high status level while others are not as highly prioritized and lapse through no fault of their own.

Thanks @JustinCappos . The discussion was also around automating the validation as much as possible, so Graduated+ doesn't require a full DD process and the load remains low.

@JustinCappos
Copy link
Contributor

JustinCappos commented Nov 28, 2024 via email

@caniszczyk
Copy link
Contributor

I need to think about this is a bit more, but instead of creating a new level, maybe we have an annual or biannual review for graduated projects to keep things simpler? We are essentially just wanting to do a health check

@alolita
Copy link
Member

alolita commented Nov 28, 2024

Adding some context here... The request to have graduated projects verify that they are maintaining stability & security guarantees and other production readiness requirements for operating in production environments came up in our TAB-TOC meeting at KubeCon NA in SLC. This verification/review could be streamlined but would be immensely useful to end users. Another factor as to why a new level (Graduation+) was explored by the TOC is because some CNCF graduated projects have also seen declining number of contributors and maintainers post graduation which is a project health concern.

@raravena80
Copy link
Contributor

Imo, rather than a new level, there could occasionally be a project health check (hopefully automated), with the outcome made public. Adding another level(s) could be confusing to the community. If the project is not healthy, steps could be recommended to make it 'healthy' again. If, after N health checks, the project doesn't improve its health, then a decision would have to be made (archive, lower level, etc.).

@dims
Copy link
Member

dims commented Nov 28, 2024 •
edited
Loading

Some useful prior art ... https://www.apache.org/foundation/board/reporting

Apache Project Management Committees (PMC) must report on their project's health and status quarterly to the Board of Directors. 
While PMCs manage their own technical affairs independently, the Board provides the oversight to ensure that projects continue to 
operate as healthy, community- and consensus-driven projects that support our mission.

@angellk
Copy link
Contributor

angellk commented Dec 2, 2024

I need to think about this is a bit more, but instead of creating a new level, maybe we have an annual or biannual review for graduated projects to keep things simpler? We are essentially just wanting to do a health check

@caniszczyk This was intended to be a biannual review for graduated projects that was conducted by the TAB instead of the TOC. Perhaps not another level - but a 'certification' or 'badge' that the project did complete and pass the biannual review.

@mrbobbytables
Copy link
Member

I thought of it like a "badge". The original idea was to have it be opt-in for projects with incentives to encourage projects to participate. Think a badge, some swag (go go swag driven development!), maybe some additional recognition in the project pavilion etc. The process SHOULD be relatively light and seen as a general health checkup with minimal work required from the projects.

If problems ARE surfaced (or frankly if a project continually opts not to participate) it's likely a sign that they do need help in some way.

@caniszczyk
Copy link
Contributor

If this is like a badge similar to https://www.bestpractices.dev than I'm all for it (or something automated via clomonitor.io)

I just want to avoid additional process where we may not need it (like a full new maturity level review)

@rochaporto
Copy link
Contributor Author

I think the original idea was to ensure end users the maturity as evaluated during graduation is still valid. It should be a lot easier to do than what a full DD requires. @mrbobbytables what do you mean by opt-in? If it's like a badge, you either have it or don't?

@mrbobbytables
Copy link
Member

I think the original idea was to ensure end users the maturity as evaluated during graduation is still valid. It should be a lot easier to do than what a full DD requires. @mrbobbytables what do you mean by opt-in? If it's like a badge, you either have it or don't?

optional - the graduated projects don't have to go through the graduated+ process if they don't want to, but it should heavily be incentivized to do so.

@craigbox
Copy link
Contributor

craigbox commented Dec 2, 2024

Here are a few thoughts and data points that may help guide this discussion:

  • “Graduated project” is a useful shorthand that end users rely on to consider an open source project externally vetted and production-ready
    • Because of that, there are strong commercial incentives to being a graduated project
    • Introducing a new level above graduation could, as Justin points out, just make that level the new “graduated”
  • Not all graduated projects have, or need, the same level of activity or maintenance
  • Some graduated projects no longer meet the requirements they had to meet at the time of their initial graduation
  • In response to events earlier this year, there was a community desire to clarify or change certain requirements for graduation level. However, there exists a CNCF principle that “[once] a project has graduated from incubation, new governance requirements cannot be imposed without consent of the project, except where legally required.”
  • The TOC has never moved a project backward, and seems reluctant to do this
  • Maintainer surveys demonstrate significant dissatisfaction in the time taken to move projects through levels (a current wait time of around 6 months for consideration of an application to join the Sandbox, and around 14 months for an application to move to incubation or graduation)

I believe almost all graduated projects have a vendor that sells some commercial or hosted version of them (except etcd, which has moved, or was moving, to be a sub-project of Kubernetes). All projects are likely to want to meet the new review bar. What happens in the case where they are told they do not?

Last year, when the topic of revising the matriculation/moving levels process came up, I floated the idea of eliminating levels entirely and replacing them with badges. However, changes to the levels concept were postponed to Phase 2, which, unfortunately, we have not yet progressed to.

I understand that the intention is not for this to be a TOC process, but more a reflection of production readiness by the TAB. However, if there were to be a desire to introduce a new level, might I suggest considering putting it between Incubating and Graduated, such as to speed up the movement between the existing levels? Consider lowering the bar for being an incubating project — putting the onus back onto the CNCF being an incubator, where projects are helped to meet these requirements. Alternatively, consider replacing Graduation with a series of badges that reflect the various different axes that an end user might consider an application (multiple vendors, regular releases, time to fix security vulnerabilities, etc).

@lizrice
Copy link
Contributor

lizrice commented Dec 5, 2024 •
edited
Loading

Given the workload on the TOC I'm pretty sure the last thing we need is another project maturity level that needs manual assessment! But an indicator of whether a (graduated) project remains as healthy as it was when it went through DD would be really useful. I understand that the End User TAB are working with LFX Insights to automate some health metrics. Would it make sense to have those automated metrics in place first, and then think about how we use them e.g. for regular re-reviews?

@tomkerkhove
Copy link
Contributor

I understand that the End User TAB are working with LFX Insights to automate some health metrics.

This so much. If this is to help give indicator to community that project is still alive and healthy, then metrics and automation should be able to help indicate that.

Would it make sense to have those automated metrics in place first, and then think about how we use them e.g. for regular re-reviews?

I would use the metrics as indicators and based on those, projects can be asked/required to get a review when there is a risk. Let's not do this on a cadence for cadence sake, let's use indicators as a trigger instead.

@hblixt
Copy link

hblixt commented Dec 9, 2024

No one wants to add unnecessary work burden to the TAB or TOC, so automation is key here. Once we have the metrics and automation in place, I would expect the additional burden here to be very low, as little should be needed as long as all the metrics for a project look good.

I think a badge that is kept up to date with a potential review if metrics fall below thresholds would meet the intent of monitoring health, provide some incentive to projects while still minimizing workload. Once the metrics and automation are in place, this shouldt be opt-in, but rather just part of the best practices expected from graduated projects.

@JustinCappos
Copy link
Contributor

JustinCappos commented Dec 9, 2024 via email

@hblixt
Copy link

hblixt commented Dec 9, 2024

If this idea goes forth, one other thing to consider is that metrics will vary a lot for healthy projects. For example, a specification project like TUF has very few updates because, well, it makes sense for a specification to rarely change. Whereas a code first project is getting features, bug fixes, etc. added all the time.
…
On Mon, Dec 9, 2024 at 11:49 AM Henrik Blixt @.***> wrote: No one wants to add unnecessary work burden to the TAB or TOC, so automation is key here. Once we have the metrics and automation in place, I would expect the additional burden here to be very low, as little should be needed as long as all the metrics for a project look good. I think a badge that is kept up to date with a potential review if metrics fall below thresholds would meet the intent of monitoring health, provide some incentive to projects while still minimizing workload. Once the metrics and automation are in place, this shouldt be opt-in, but rather just part of the best practices expected from graduated projects. —

Agreed, I didnt mean that we should apply the same thresholds across projects, I would hope we can make the automation smart enough to take individual project trends and uniqueness etc into account rather than trying to see all metrics as equal across projects.

@craigbox
Copy link
Contributor

Can the End User TAB confirm that all their concerns would be allayed with only the use of metrics?
"Project no longer offers stable builds" would not have; for example; at least, not without adding a new metric of "when was the last stable build" 😄

@rochaporto
Copy link
Contributor Author

rochaporto commented Dec 10, 2024 •
edited
Loading

Trying to summarize the discussion, there seem to be two main points.

  • Should be opt-in: i'm not sure what we mean by this... moving levels is already opt-in, projects explicitly request to move levels no one is forcing them. In the same way, projects could decide to make the effort to be/remain graduated+ or not, stating this as opt-in is confusing.

  • Level vs badge: as an end user this makes little to no difference - if we would manage to fully automate the process from sandbox to incubation, an incubation badge would be enough. As an end user, graduated+ brings the value of saying that the graduation criteria is still valid (verified usage in production by multiple adopters, openssf best practices and other badges still valid, etc, etc). It should be much lighter than graduation, but i have doubts we can fully automate it in a badge today - definitely a good start.

@mcderk
Copy link

mcderk commented Dec 11, 2024

What if we implemented a ‘verified’ checkmark to indicate that a health review was completed within the past 1–2 years?

This checkmark could display the date of the review and include a link to the health check report as a point-in-time reference. This approach avoids adding unnecessary complexity by introducing a new level or term like graduated+.

Over time, the process could potentially be automated to ensure efficiency and consistency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rochaporto rochaporto

@kgamanji kgamanji

Labels
enhancement
Projects
CNCF TOC Board
Status: New
Milestone
No milestone
Development

No branches or pull requests