gRPC graduation application

[srini100]

This is a proposal to consider gRPC for CNCF graduation.

[mattklein123]

@srini100 thanks for this proposal.

In terms of impact and production use, gRPC clearly qualifies for graduation from my perspective. I have a few questions about governance and language support however:

1. Although I see 2 maintainers on https://github.com/grpc/grpc/blob/master/MAINTAINERS.md that don't work at Google, about 45 do work at Google. Can you describe the governance and design review process? Is it meaningfully possible for changes to gRPC core to be implemented without Google's consent? As a concrete example, recently gRPC decided to adopt Envoy's xDS protocol for lookaside load balancing. Was this proposal reviewed by the community or was it decided internally within Google?

2. As I like to say, and I mean no direct offense by this statement, the magic behind gRPC is in large part due to the magic in proto3. Given that protobuf itself is not in a foundation and is controlled by Google, can you comment on the interplay between gRPC and protobuf as it relates to governance? Speaking on behalf of end-users, it has been difficult at times to get the gRPC and protobuf teams to acknowledge end user pain outside of Google (a good example of this lead to the creation of https://github.com/envoyproxy/protoc-gen-validate).

3. Language support outside of the core supported languages by Google (C++/Java/Go) has traditionally been extremely haphazard. AFAICT this is the primary area in which non-Google contributions have come in. Can you comment on the current state of non-core languages, how you see the project evolving production ready support for these languages, and community involvement? This is somewhat related to (1) given that in some cases quality language support may require changes to gRPC core.

Thank you!

cc @jbeda

[srini100]

@mattklein123, thanks for your review and questions. Please see my responses below:

1. Although I see 2 maintainers on https://github.com/grpc/grpc/blob/master/MAINTAINERS.md that don't work at Google, about 45 do work at Google. Can you describe the governance and design review process? Is it meaningfully possible for changes to gRPC core to be implemented without Google's consent? As a concrete example, recently gRPC decided to adopt Envoy's xDS protocol for lookaside load balancing. Was this proposal reviewed by the community or was it decided internally within Google?

gRPC governance process is described here. This was recently updated to have a well defined process for anyone to become a maintainer if they meet the described criteria (similar to other OSS projects). We are already making use of this new process to consider the request from a contributor from Skyscanner to become a maintainer of gRPC Core repo. We hope this will encourage more gRPC users to consider becoming maintainers. Over the years, many large contributions from different companies have been accepted.

The design and feature proposals are done on Github via the gRFC process which has been in place since Jan’17. Over 80 proposals have been made since then. Sometimes the initial ask may come from a user on another forum and one of the maintainers will work with them and make a proposal. Some examples of input from non-Google contributors are Service Config in DNS, C++ wrapping, custom transports and Interceptors in C#, NodeJS and Ruby. Most recently, Skyscanner, Dropbox and Google have been collaborating on Python Asyncio gRFC. This is a good example where a new set of APIs and features in gRPC Core repo are being driven by companies other than Google. Some PRs (1, 2, 3) are already merged. Other example PRs are public resolver API in gRPC Core and SSL session caching. So yes, if a non-Google contributor shows expertise, motivation and commitment and there are other non-Google reviewers with sufficient expertise, Google consent is not required. But, given the large number of committers from Google, it is natural that someone from Google will help with the reviews and influence the design and implementation.

Regarding the adoption of xDS APIs, the intention to move to xDS was announced on grpcio and in a gRFC proposal. The decision was based on user friction with the existing gRPC LB solution that was suffering from being proprietary and lacking open source control plane implementation. This enables a better ecosystem play for gRPC with other CNCF and open source projects and reduces dependence on Google for driving the roadmap.

2. As I like to say, and I mean no direct offense by this statement, the magic behind gRPC is in large part due to the magic in proto3. Given that protobuf itself is not in a foundation and is controlled by Google, can you comment on the interplay between gRPC and protobuf as it relates to governance? Speaking on behalf of end-users, it has been difficult at times to get the gRPC and protobuf teams to acknowledge end user pain outside of Google (a good example of this lead to the creation of https://github.com/envoyproxy/protoc-gen-validate).

It is well known that gRPC and protobuf go together very well but gRPC is designed to work with other IDLs and serialization formats and similarly protobuf can be used with other communication protocols too. We take extra care in not taking a dependency on protobuf in gRPC core (example) and keeping gRPC independent of protobuf. We published a blog to show how easy it is to use JSON with gRPC. Microsoft has also published their serialization protocol Bond over gRPC.

Regarding the governance, these are treated as independent projects. Committers/maintainers are different and have good cordial relationship. Just like gRPC has interaction with Netty and now with Envoy Data Plane APIs, it has interaction with Protocol Buffers. Most of the user friction with protobuf is unfortunately around protobuf tooling which is outside the scope of gRPC project. gRPC & Protobuf teams acknowledge the user pain. Protobuf related tools is a great place to start community contributions. It is orthogonal to gRPC project, but as a part of Protobuf community efforts, the idea of starting community efforts to improve protobuf tooling is very appealing.

3. Language support outside of the core supported languages by Google (C++/Java/Go) has traditionally been extremely haphazard. AFAICT this is the primary area in which non-Google contributions have come in. Can you comment on the current state of non-core languages, how you see the project evolving production ready support for these languages, and community involvement? This is somewhat related to (1) given that in some cases quality language support may require changes to gRPC core.

Besides C++, Java and Go, Google maintainers are also fully supporting Python, Ruby, Node, ObjectiveC, C# and PHP. All these languages are on par with core languages in terms of features and version releases. This shows that Google is committed to supporting more languages than just the core languages, but of course, Google maintainers themselves cannot develop gRPC for a number of other languages that the community is interested in. To have gRPC available in more languages, we rely on community contributions which do not require Google consent. For example, gRPC-Swift is being jointly developed by Apple, Lyft, Timing and Google. Currently, it relies on gRPC C-core but soon it will move to Swift-NIO based stack for production readiness. gRPC-Dotnet is another example where contributions from Microsoft has enabled production use of gRPC in ASP.NET Core. This was announced in .NET Conf recently. Even for existing languages, we are happy to accept large community contributions like the Python Ayncio I mentioned above and Interceptors in C#, NodeJS and Ruby. For a language support to go GA we will ensure that it passes interop tests with other implementations and a sustainable test infrastructure is available for long term support.

Thanks!

[srini100]

Updated the "security audit" section with details of a third-party security audit that was recently completed.
Also, added Skyscanner as one of the maintainers of gRPC Core repo.

[caniszczyk]

Who wants to volunteer from the @cncf/toc to do the due diligence for gRPC?

@bgrant0607 ?

[ultrasaurus]

Should due diligence be prepared by someone who doesn't work for Google? There may be a perceived conflict of interest.

Perhaps a TOC Contributor could draft due dilligence or someone from the group who part of proposed SIG-Networking (see PR #317)

[xiang90]

happy to help if TOC from Google is a concern.

[thisisnotapril]

hi all; would love to get this moving - any update?

[xiang90]

hi all; would love to get this moving - any update?

I am happy to help on the DD. Usually the project owners + the CNCF SIG will make a draft. And the TOC member will give it a review, and "approve" it. If everything is OK, the voting process will begin.

The DD template is here: https://github.com/cncf/toc/blob/master/process/dd-review-template.md

Here are some recent DD examples:

1. Falco: https://docs.google.com/document/d/1TJCzW8dQ6858lw2UNY-H5LMnvEd4GzwjuOcDInimeyA/edit?usp=sharing

2. Harbor: https://docs.google.com/document/d/15gX7EeeXQThEvVMGpL-0a1mOwGuByLtMfvXNJaKT0A0/edit?usp=sharing

3. Vitess: https://docs.google.com/document/d/1TDlRdgfTiEWunpav-G8gkaQF7Zk84-9tNAXyv1I0Kws/edit?usp=sharing

[amye]

Assigning to @justincormack for review, keeping open.

[fjcrabiffosse]

Is there an ETA for gRPC graduation? Latest commit of the graduation application dates from September 2019. What's keeping the graduation from proceeding forward?

[amye]

Is there an ETA for gRPC graduation? Latest commit of the graduation application dates from September 2019. What's keeping the graduation from proceeding forward?

@justincormack is working with the project on due diligence. This is moving forward.

[tomkerkhove]

Is there an update on this? I'm curious why gRPC has not completed this given its obvious position in the market today.

[mattfarina]

The TOC met with maintainers from the gRPC project, today, to discuss the graduation application that has been open for more than 3.5 years due to a variety of outstanding reasons. In the meeting we discussed a path forward and the TOC gave some concrete feedback. The goal was to give the maintainers of gRPC what they needed to unblock them and enable them to progress towards graduation.

The following path forward was discussed:

1. The current PR with the graduation application will be closed. The information is out of date. Note, when a new graduation application is created this PR can be referenced to link to the history.
2. gRPC maintainers will meet with TAG Contributor Strategy to do a governance review in order to get feedback and suggestions on improving it.
3. gRPC will update their governance to address the issues.
4. Once gRPC maintainers have the governance in place and can show they are following it the project can open a new PR with a graduation application.

During the meeting the primary issue discussed related to the documented governance and how it is followed. The feedback included:

• There are sections of the governance that do not appear to be followed (e.g., how maintainers are added)
• The section on someone no longer being a maintainer has opportunities to be gamed and is not very explicit.
• Some codebases do not have documented maintainers
• The governance calls out "senior maintainers" and the proposals have a11r in a special role. But, there is not documented governance around this.

On the whole, the governance should document what's happening (e.g., the different roles and processes around them) . The governance should then be followed.

The issue of committers from multiple companies came up. The organization/project has maintainers from other companies (e.g., Microsoft and Apple on some repos). We discussed governance and intent around the multiple committers graduation criteria with suggestions on how to proceed.

We, the TOC, believe the outstanding issues are now unblocked and gRPC has a path forward to progress toward graduation.