Add clarification to CNCF graduation criteria

[cathyhongzhang]

Current CNCF graduation criteria do not lay out everything the TOC looks at. For example, TOC will ask the project leads to reach out to TAG-security for a security check and get their recommendation/input, but this is not documented in the graduation criteria or graduation DD. TOC is also discussing whether to leverage TAG Contributor Strategy's expertise to help review the project's governance to make sure it is open and neutral. Without clearly spelling out these, there could be confusion among the project maintainers. We have seen some stumbling along this application journey. To set the project maintainers up well for the graduation review process and avoid potential insistency/controversy about the evaluation/approval/rejection result, it is worth adding such clarification to the graduation criteria/DD document. Thoughts?

[TheFoxAtWork]

related #907

Some things to consider when updating:

• availability of these groups to take on additional work
• somethings dont need to be criteria for graduation - they can be things we've seen highly successful projects do and would strongly recommend others follow
• project Due Diligence is often project specific - the current happenings and challenges in one project's process may look entirely different than another- we should plan for the common areas and provide guidance (perhaps) on domain or tech specifics.

[xmulligan]

Slightly related, during Cilium's graduation process, we have also found that the document structure is different than in the repo, we based ours off of Argo and Flux.

The general structure is

• Background
• Project Goal
• Current Status
• Project Progress since Incubation
• Future Plans
• Project Scope
• Value-add to the CNCF Ecosystem
• Alignment with other CNCF projects
• Formal Requirements
• Other Considerations

[xmulligan]

Would it be helpful to create a DD template related to https://github.com/cncf/toc/blob/main/process/due-diligence-guidelines.md?

[tomkerkhove]

As a maintainer going through a proposal now, that would definitely be useful and can be hosted on Google Docs thanks to #980

[rochaporto]

One more item to clarify on the graduation criteria:

Projects can remain in an incubating state indefinitely, but they are normally expected to graduate within two years.

We might want to clarify expectations here.

[cathyhongzhang]

Thanks all for the good comments/suggestions! Everyone, you are welcome to share your questions/frustration about the application to the graduation process and provide suggestions!

[xmulligan]

It is also currently not documented in the TOC repo if a TAG Security review is required. https://github.com/cncf/tag-security/tree/main/assessments

[craigbox]

It is also currently not documented in the TOC repo if a TAG Security review is required.

Would some sort of overview security review make more sense as part of an incubation requirement? The TOC already asks projects that are applying for graduation to have undergone a third-party security review.

[tomkerkhove]

Agreed; we now have to go through 2 security reviews.

[tomkerkhove]

Looking at the process for this TAG Security review; this is going to take another bunch of weeks as well. If this is a strong requirement, then I believe both the external + TAG review should be merged to optimize the time of all parties (CNCF + TAG + Security vendor + Maintainers).

[tomkerkhove]

Another one I start seeing pop up on #997 are:

• First passing then Silver OpenSSF Best Practice Badge
• Project provides scalability benchmarks and sets expectations around scaling

[TheFoxAtWork]

cross posting the comment on security reviews from the PR here: #997 (comment)

...structure it [TAG Security Review process] such that projects may complete portions of the overall "Security Package" aligned with where they are at in the CNCF "lifecycle". This revision was designed to prevent the security review (which often doesn't have enough reviewers and as a result does take longer) from becoming a bottleneck for a project pursuing graduation. If projects completed the "self-assessment" during Sandbox, then optional "joint-review" during Incubation (well before considering graduation) these two security documents form a security package that can be leveraged by Security Auditors to assist in the formal audit.

[tomkerkhove]

Looking at the process for this TAG Security review; this is going to take another bunch of weeks as well. If this is a strong requirement, then I believe both the external + TAG review should be merged to optimize the time of all parties (CNCF + TAG + Security vendor + Maintainers).

As follow-up, and I'll quote @TheFoxAtWork from cncf/tag-security#1030:

@tomkerkhove thank you so much for submitting this issue! Since KEDA has applied for graduation, the project should seek a security audit through the Service Desk, as the final audit will supersede the security review. I would recommend - if the project is interested - request a security pal to assist with a lightweight threat model prior to the audit.

So let's clarify this in the criteria as this was a bit of a confusing aspect for a proposing project.

[cathyhongzhang]

@tomkerkhove yes, we need to clarify this in the criteria.

[craigbox]

I think it would be worthwhile to have more definition around the need for, or use of, a due diligence document, at the graduation stage.

The process for incubation is well understood, and experience generally maps to that published guidance. A proposal and Due Diligence are both clearly stated as being required, and both a template and guidelines for due diligence are provided.

(The graduation criteria states that the incubation phase is where "full" due diligence is done, but the proposal documentation says that "a majority" of due diligence happens at the incubation stage.)

The documented process for graduation phase only directly refers to the requirement for a proposal, and provides a template which lists six "criteria", with the seventh being that the TOC vote on whether the six are met. That word may not have been chosen deliberately, but it means "a principle or standard by which something may be judged or decided", and so suggests to me that graduation is a matter of meeting the defined criteria.

If the TOC is looking at things that are not in the criteria, then I am in strong support of updating the criteria to match. For example, the proposal guidelines state "The proposal addresses how the project has grown since incubation and any concerns from incubation DD in addition to the standard graduation requirements." Must a project have grown since incubation? Is this now part of the criteria?

If there is a requirement to publish a new due diligence document in order for a project to graduate, could that requirement please be added concretely, and a template provided?

[cathyhongzhang]

@craigbox I agree with your points. We will consider your input when updating the document

[TheFoxAtWork]

other considerations from various discussions to be evaluated as part of this:

• projects have a healthy distribution of commit author company/organization diversity, targeting less than 40% contributions from a single company/org.
• The project contains governance and structure such that a single company or organization may NOT achieve majority in the direction, technical, and governance decisions of the project.

[craigbox]

Tying graduation to a “healthy distribution of commit author company/organization diversity” may signal a change of intentionality about the way CNCF presents its projects which I think warrants larger discussion.

Modern open source is almost all written by people who are paid to do so. For the top company to have <40% of the commits to a project, you would realistically need to have at least three equal-sized contributors to the project. That counts out having a project that is largely maintained by a single vendor being a “Graduated” project, and thus receiving the full-throated recommendation of the CNCF.

Worse, as Liz points out, a perverse incentive for a vendor would be to drop their contribution in order that other contributors catch up, or to “launder” their commits through another organization.

Many graduated projects don’t come close to the 40% bar today. Would they be asked to give up their status? For example:

• 96% of Linkerd commits are from Buoyant
• 71% of TiKV commits are from PingCAP
• 78% of Harbor commits are from VMware
• 66% of OPA commits are from Styra
• 56.2% of commits to Vitess are by Planetscale, with another 47% being from Google/YouTube: the company that developed Vitess before most of the Vitess team left to found Planetscale. In this case, the individuals seem more important than the employer; our community has a history of people leaving a company but staying with a project. (Also, the fact the numbers sum to more than 100% suggests that commits have multiple authors and Devstats is probably not a suitable tool for doing this math).

---

From #997:

We also don't want to have a single entity (company or organization or individual) be the majority decision maker of a projects technical direction, if that entity is acquired or disappears, the project can be left out to dry, and in some case unable to recover because the primary pool of contribution has evaporated.

This is a reality of all software, and open source itself is commonly offered as a hedge against this eventuality.

A thought exercise: should the CNCF recommend that people use gRPC, which is developed and depended on by one of the five largest companies in the world, or should it prefer an alternative which is maintained equally by one person from three different companies, in their evenings?

This change would also rule out much governance based on maintainership, which is generally conferred on those who have contributed the longest/the most.

---

I think a lot of the debate here is because of how the CNCF positions and promotes its projects. Right now, I interpret “incubating” and “graduated” as the CNCF presents them: “a signal by CNCF as to what sorts of enterprises should be adopting different projects”. Companies are founded with all their code submitted to the CNCF Sandbox as soon as possible. Projects want to be graduated.

If the opinion of this TOC is that, to be recommended to pragmatists, a project must have diverse authorship and maintainership, then I believe that would necessitate a major change in the CNCF’s focus, to being an organization which helps projects achieve this bar.

Alternatively, why not consider moving away from a single track? Participants come and go over time, so it does not make sense to have recommendation status be a lifetime achievement. For example, what about simply having “CNCF Projects”, and publishing their current security status and company contribution diversity somewhere as tags or badges?

[cathyhongzhang]

I like the idea of "publishing current security status and company contribution diversity somewhere as tags" to the graduation projects. We can get such information for each project during its annual review. But we still need some high bars for a project to be marketed as a CNCF-endorsed mature project with good support when a user adopts it. These project maturity levels are useful to end users when they evaluate whether to adopt a project or not.

[craigbox]

Are "sandbox" and "incubating" (or their renamed equivalents) good enough indicators of "immature" and "mature"?

[cathyhongzhang]

interesting question. This is a fundamental change that may impact many existing projects in different maturity statuses as well as those projects in the TOC review pipeline to change status. Let's discuss this in the TOC meeting tomorrow.

[prithvi1307]

Any progress on this? Are we discussing this issue further?

[TheFoxAtWork]

Hi @prithvi1307 ! This work is going to be executed under the TAG CS issue linked above (cncf/tag-contributor-strategy#366). Additional feedback, suggestions, call outs, and interest in participation are being captured on that issue. The TOC is currently working on defining the scope of the group and once we have that the TAG CS issue will be updated. Apologies for the delay in resolving this issue, we're juggling several items in our backlog with the availability we have currently.

Did this answer your question?

[duglin]

@TheFoxAtWork is there a set of regularly scheduled meetings that we can attend to see (and be part of) the conversations? Aside from one call shortly after the initial TOC meeting we had a long time ago I haven't noticed any other discussions but I may have missed some threads.

[TheFoxAtWork]

No meetings scheduled as of yet. The conversations thus far are only what is on the TAG CS issue and among the TOC. The comments on the TAG CS issue are the capture of observations as TOC members engage with projects conducting activities (such as Due Diligence).

[xmulligan]

I guess this can be closed with #1263 right?

[TheFoxAtWork]

Yes thank you @xmulligan !