-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
241 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,241 @@ | ||
| --- | ||
| title: Graduate CustomResourceDefinitions to GA | ||
| authors: | ||
| - "@jpbetz" | ||
| - "@roycaihw" | ||
| - "@sttts" | ||
| owning-sig: sig-api-machinery | ||
| participating-sigs: | ||
| - sig-api-machinery | ||
| - sig-architecture | ||
| reviewers: | ||
| - "@deads2k" | ||
| - "@lavalamp" | ||
| - "@liggitt" | ||
| - "@mbohlool" | ||
| - "@sttts" | ||
| approvers: | ||
| - "@deads2k" | ||
| - "@lavalamp" | ||
| editor: TBD | ||
| creation-date: 2018-04-15 | ||
| last-updated: 2018-04-24 | ||
| status: provisional | ||
| see-also: | ||
| - [Umbrella Issue](https://github.com/kubernetes/kubernetes/issues/58682) | ||
| - [Vanilla OpenAPI Subset Design](https://docs.google.com/document/d/1pcGlbmw-2Y0JJs9hsYnSBXamgG9TfWtHY6eh80zSTd8) | ||
| - [Pruning for CustomResources KEP](https://github.com/kubernetes/enhancements/pull/709) | ||
| --- | ||
|
|
||
| # Title | ||
|
|
||
| Graduate CustomResourceDefinitions to GA | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| * [Summary](#summary) | ||
| * [Motivation](#motivation) | ||
| * [Goals](#goals) | ||
| * [Non-Goals](#non-goals) | ||
| * [Proposal](#proposal) | ||
| * [Defaulting and pruning for custom resources is implemented](#defaulting-and-pruning-for-custom-resources-is-implemented) | ||
| * [CRD v1 schemas are restricted to a subset of the OpenAPI specification](#crd-v1-schemas-are-restricted-to-a-subset-of-the-openapi-specification) | ||
| * [Generator exists for CRD Validation Schema v3 (Kubebuilder)](#generator-exists-for-crd-validation-schema-v3-kubebuilder) | ||
| * [CustomResourceWebhookConversion API is GA ready](#customresourcewebhookconversion-api-is-ga-ready) | ||
| * [CustomResourceSubresources API is GA ready](#customresourcesubresources-api-is-ga-ready) | ||
| * [v1 API](#v1-api) | ||
| * [Test Plan](#test-plan) | ||
| * [Integration tests for GA](#integration-tests-for-ga) | ||
| * [e2e tests for GA](#e2e-tests-for-ga) | ||
| * [Conformance plan for GA](#conformance-plan-for-ga) | ||
| * [Scale Targets for GA](#scale-targets-for-ga) | ||
| * [Graduation Criteria](#graduation-criteria) | ||
| * [Post-GA tasks](#post-ga-tasks) | ||
| * [Arbitrary subresources as Post-GA task](#arbitrary-subresources-as-post-ga-task) | ||
| * [Implementation History](#implementation-history) | ||
|
|
||
| ## Summary | ||
|
|
||
| CustomResourceDefinitions (CRDs) are the way to extend the Kubernetes API to | ||
| include custom resource types that behave like the native resource types. CRDs | ||
| have been in Beta since Kubernetes 1.7. This document outlines the required | ||
| steps to graduate CRDs to general availability (GA). | ||
|
|
||
| ## Motivation | ||
|
|
||
| CRDs are in widespread use as a Kubernetes extensibility mechanism and have been | ||
| available in beta for the last 5 Kubernetes releases. Feature requests and bug | ||
| reports suggest CRDs are nearing GA quality, and this KEP aims to itemize the | ||
| remaining improvements to move this toward GA. | ||
|
|
||
| ### Goals | ||
|
|
||
| Establish consensus for the remaining essential improvements needed to move CRDs to GA. | ||
|
|
||
| Guiding principles: | ||
| * if a missing feature forces or encourages users to implement non-Kubernetes-like APIs, and therefore damages the ecosystem long term, it belongs on this list. | ||
| * If a feature cannot be added as an after-thought of a GA API, it or some preparation belongs on this list. | ||
|
|
||
|
|
||
| ### Non-Goals | ||
|
|
||
| Full feature parity with native kubernetes resource types is not a GA graduation goal. See above guiding principles. | ||
|
|
||
| ## Proposal | ||
|
|
||
| Objectives to graduate CRDs to GA, each of which is described in more detail further below: | ||
|
|
||
| * Defaulting and pruning for custom resources is implemented | ||
| * CRD v1 schemas are restricted to a subset of the OpenAPI specification (and there is a v1beta1 compatibility plan) | ||
| * Generator exists for CRD Validation Schema v3 (Kubebuilder) | ||
| * CustomResourceWebhookConversion API is GA ready | ||
| * CustomResourceSubresources API is GA ready | ||
|
|
||
| Bug fixes required to graduate CRDs to GA: | ||
|
|
||
| * See “Required for GA” issues tracked via the [CRD Project Board](https://github.com/orgs/kubernetes/projects/28). | ||
|
|
||
| For additional details on already completed features, see the [Umbrella Issue](https://github.com/kubernetes/kubernetes/issues/58682). | ||
|
|
||
| See [Post-GA tasks](#post-ga-tasks) for decided out-of-scope features. | ||
|
|
||
| ### Defaulting and pruning for custom resources is implemented | ||
|
|
||
| Both defaulting and pruning and also read-only validation are blocked by the | ||
| OpenAPI subset definition (next point). An update of the [old Pruning for | ||
| CustomResources KEP](https://github.com/kubernetes/enhancements/pull/709) and the implementation | ||
| ([pruning PR](https://github.com/kubernetes/kubernetes/pull/64558), [defaulting | ||
| PR](https://github.com/kubernetes/kubernetes/pull/63604)), are follow-ups as soon as unblocked. | ||
|
|
||
| ### CRD v1 schemas are restricted to a subset of the OpenAPI specification | ||
|
|
||
| See [Vanilla OpenAPI Subset Design](https://docs.google.com/document/d/1pcGlbmw-2Y0JJs9hsYnSBXamgG9TfWtHY6eh80zSTd8) | ||
|
|
||
| ### Generator exists for CRD Validation Schema v3 (Kubebuilder) | ||
|
|
||
| See [kubebuilder’s | ||
| crd-gen](https://github.com/kubernetes-sigs/controller-tools/tree/master/cmd/crd) | ||
| and a more general | ||
| [crd-schema-gen](https://github.com/kubernetes-sigs/controller-tools/pull/183), | ||
| to be integrated into kubebuidler’s controller-tools. | ||
|
|
||
| ### CustomResourceWebhookConversion API is GA ready | ||
|
|
||
| Currently conversion webhooks are v1alpha1. Our goal is to move to v1beta1 in | ||
| the Kubernetes 1.15 release, and then graduate this to GA as part of the CRD to | ||
| GA graduation; we won't track this as a separate feature from CRDs once it's in | ||
| v1beta1. | ||
|
|
||
| Conversion webhooks to beta tasks: | ||
| - ConversionReview API v1 (incl. fields that need to be updated for v1beta1 or v1) | ||
| - Add conversion webhook tests enumerated in below [Test Plan](#test-plan) | ||
| * Add comprehensive metrics to conversion webhooks | ||
|
|
||
| ### CustomResourceSubresources API is GA ready | ||
|
|
||
| Currently custom resource subresources are v1beta1 and provide support for the | ||
| /status and /scale subresources. We plan to graduate this to GA as part of the | ||
| CRD to GA graduation. See [Arbitrary subresources as Post-GA | ||
| task](#arbitrary-subresources-as-post-ga-task) details on post GA tasks. | ||
|
|
||
| ### v1 API | ||
|
|
||
| The CRD `v1` API will be the same as the `v1beta1` but with all changes to the API from the GA tasks: | ||
|
|
||
| * Rename misnamed json field [JSONPath](https://github.com/kubernetes/kubernetes/blob/06bc7e3e0026ea25065f59f4bd305c0b7dbbc145/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/types.go#L226-L227) to `jsonPath` | ||
| * [Replace top-level fields with per version fields](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/customresource-conversion-webhook.md#top-level-fields-to-per-version-fields) | ||
| * Restrict OpenAPI per [Vanilla OpenAPI Subset Design](https://docs.google.com/document/d/1pcGlbmw-2Y0JJs9hsYnSBXamgG9TfWtHY6eh80zSTd8) | ||
|
|
||
| ## Test Plan | ||
|
|
||
| ### Integration tests for GA | ||
|
|
||
| TODO: complete this list | ||
|
|
||
| * [Pruning for CustomResources KEP](https://github.com/kubernetes/enhancements/pull/709) | ||
| * TODO | ||
| * [Vanilla OpenAPI Subset Design](https://docs.google.com/document/d/1pcGlbmw-2Y0JJs9hsYnSBXamgG9TfWtHY6eh80zSTd8) | ||
| * TODO | ||
| * CRD versioning/conversion (https://github.com/kubernetes/kubernetes/issues/64136): | ||
| * Ensure what is persisted in etcd matches the storage version | ||
| * Set up a CRD, persist some data, changed the version, and ensure the previously persisted data is readable | ||
| * Ensure discovery docs track a CRD through creation, version addition, version removal, and deletion | ||
| * Ensure `spec.served` is respected | ||
|
|
||
| ### e2e tests for GA | ||
|
|
||
| TODO: complete this list | ||
|
|
||
| * Ensure v1beta1 and v1 CRDs are round trippable | ||
| * Ensure CR watch observes events on create/update/delete | ||
| * Ensure a CR watch is correctly terminated if CRD is deleted | ||
|
|
||
| ### Conformance plan for GA | ||
|
|
||
| TODO: complete this list | ||
|
|
||
| * Promote existing e2e tests: | ||
| * [CustomResourceDefinition Watch](https://github.com/kubernetes/kubernetes/blob/36ccff1b27b6aee5c5599334842cb2b8cf8063f3/test/e2e/apimachinery/crd_watch.go#L37) | ||
| * [CustomResourcePublishOpenAPI](https://github.com/kubernetes/kubernetes/blob/36ccff1b27b6aee5c5599334842cb2b8cf8063f3/test/e2e/apimachinery/crd_publish_openapi.go#L52) | ||
| * [CustomResourceConversionWebhook](https://github.com/kubernetes/kubernetes/blob/36ccff1b27b6aee5c5599334842cb2b8cf8063f3/test/e2e/apimachinery/crd_conversion_webhook.go#L81) | ||
|
|
||
| ### Scale Targets for GA | ||
|
|
||
| * TODO quantify: Read/write latency of CRDs within X% of native Kubernetes types | ||
| * TODO quantify: Latency degrades less than X% for up to 100k Custom Resources per CRD kind | ||
| * TODO quantify: Webhook conversion QPS of a noop converter is within X% of QPS with no webhook | ||
| * Coordinate with sig-scalability | ||
|
|
||
| ## Graduation Criteria | ||
|
|
||
| To mark these as complete, all of the above features need to be implemented. An | ||
| umbrella issue is tracking all of these changes. Also there need to be | ||
| sufficient tests for any of these new features and all existing features and | ||
| documentation should be completed for all features. | ||
|
|
||
| See [umbrella issue](https://github.com/kubernetes/kubernetes/issues/58682) for status. | ||
|
|
||
| ## Post-GA tasks | ||
|
|
||
| See the [umbrella issue](https://github.com/kubernetes/kubernetes/issues/58682) for details on Post-GA tasks. | ||
|
|
||
| ### Arbitrary subresources as Post-GA task | ||
|
|
||
| Currently CRDs have support for [status and scale | ||
| subresources](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#subresources). | ||
| These subresources are enabled by including them in the `subresources` CRD field, e.g.: | ||
|
|
||
| ```yaml | ||
| apiVersion: apiextensions.k8s.io/v1beta1 | ||
| kind: CustomResourceDefinition | ||
| ... | ||
| spec: | ||
| ... | ||
| subresources: | ||
| status: {} | ||
| scale: | ||
| specReplicasPath: .spec.replicas | ||
| statusReplicasPath: .status.replicas | ||
| labelSelectorPath: .status.labelSelector | ||
| ``` | ||
| Arbitrary subresources could be declared in the same way. | ||
| ```yaml | ||
| apiVersion: apiextensions.k8s.io/v1beta1 | ||
| kind: CustomResourceDefinition | ||
| ... | ||
| spec: | ||
| ... | ||
| subresources: | ||
| foo: | ||
| <subresource configuration> | ||
| ``` | ||
| The subresource behavior would be defined by the fields under | ||
| `.spec.subresource.<subresource-name>`. It could define the resource behavior | ||
| directly, or define how a webhook or aggregated apiserver could serve the subresource. | ||
|
|
||
| ## Implementation History | ||
|
|
||
| TODO |